lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 2.7458 seconds; current memory usage: 0.178 Gibyte, delta: 0.040 Gibyte, peak delta: 0.052 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0069 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0105 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/pipeline_orig.qgraph DEEP/20220821/B1m/science#step1/20240307T033644Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.0164 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 13.7406 seconds; current memory usage: 0.367 Gibyte, delta: 0.189 Gibyte, peak delta: 0.177 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.367 Gibyte (main), 0.358 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0196 seconds; current memory usage: 0.367 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.367 Gibyte (main), 0.358 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 744 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220821_B1m_science#step1_20240307T033644Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.1492 seconds; current memory usage: 0.367 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.367 Gibyte (main), 0.358 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 746 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2823 seconds; current memory usage: 0.369 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.369 Gibyte (main), 0.358 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: e15f6fd2-423c-4133-9782-1c97bf72e91a parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 03:37:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 03:37:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:37:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 03:37:25 proc_lsst.multi:170 [INFO] [multi] job_id 4364 proc_lsst.multi INFO: [multi] job_id 4364 2024-03-07 03:37:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:25 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 4364 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 4364 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 4364 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 7a10b893c16f727d279b81013562d959 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/deffdf84-00f5-42ac-8b35-fbbf1e104a66_isr_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/deffdf84-00f5-42ac-8b35-fbbf1e104a66_isr_1124087_18.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 48b254d78dcb253b4cebf5f9e381ddc0 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/d2c2718b-8f73-4be9-8e85-8e62e63eb887_isr_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/d2c2718b-8f73-4be9-8e85-8e62e63eb887_isr_1123958_58.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash e08887d3dd37a6cf7546eca6c91b03f4 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/56de81a0-049f-4c03-af45-f0dd1a38afdb_isr_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/56de81a0-049f-4c03-af45-f0dd1a38afdb_isr_1123956_3.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 0d921c308174a5050f59785c7bb61231 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/ef4efb41-09ad-4c42-906c-1d1c6b84e5d0_isr_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/ef4efb41-09ad-4c42-906c-1d1c6b84e5d0_isr_1123958_11.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 1670d30648702843e2f9e90dc67dc7a2 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f41214ad-b66b-426d-b5e4-1c7a6ca49c89_isr_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f41214ad-b66b-426d-b5e4-1c7a6ca49c89_isr_1123958_1.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash c28b9bd8f98819dd33a4de39eca78cb5 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65435dce-3ca4-48d1-90a5-8fe0851e1192_isr_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65435dce-3ca4-48d1-90a5-8fe0851e1192_isr_1123956_32.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 4e5fb40538fbc1430356832a09d982b1 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1a90975e-bcf4-44a0-a9f8-8ee889807b33_isr_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1a90975e-bcf4-44a0-a9f8-8ee889807b33_isr_1124087_54.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 6c5351e27fabc4678bd89fa5c922d32b parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/626d844d-b14b-444a-97d4-e8d9993815e0_isr_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/626d844d-b14b-444a-97d4-e8d9993815e0_isr_1123958_17.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 5fcf0adb39bc05a967e8ee6366f47811 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/961418db-2d8c-4744-930f-036a28ffde04_isr_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/961418db-2d8c-4744-930f-036a28ffde04_isr_1124087_3.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash c5c420fbe6e8ad1514f316754a8f3b01 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dfd1ea59-1ae0-4b08-9044-221861ee6fd7_isr_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dfd1ea59-1ae0-4b08-9044-221861ee6fd7_isr_1123956_2.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 258b2937a8dca9c14e7d14b8eef57c27 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/02f96563-44f0-415b-874c-374712afd587_isr_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/02f96563-44f0-415b-874c-374712afd587_isr_1123958_57.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 4476d8cdf3102144b1cbb64dbcc17d15 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bda18c61-7e4c-414d-9027-bbd0420675c9_isr_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bda18c61-7e4c-414d-9027-bbd0420675c9_isr_1124087_37.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 275b76eb4e1ad94cdd2c61f56d0d8256 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9ceab93-00d6-4de2-977e-2535031b1926_isr_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9ceab93-00d6-4de2-977e-2535031b1926_isr_1123956_27.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 9a6f2bffe028a51ea152729bb0daeb98 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/90911a30-cf15-4550-ae21-05770920adcc_isr_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/90911a30-cf15-4550-ae21-05770920adcc_isr_1124087_43.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 4bd6afa36e946c3c32d6db8663077af0 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/86db1407-c4a2-49a2-9d4b-90ddba3e1343_isr_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/86db1407-c4a2-49a2-9d4b-90ddba3e1343_isr_1123958_45.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash f6848497815d68859de1bb6606b166c4 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a155b02c-df2d-4cf5-8398-9f710641f0a7_isr_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a155b02c-df2d-4cf5-8398-9f710641f0a7_isr_1123954_61.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 901ed7baaaa98ec328d167e600a4af97 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/23110368-8bbf-40dc-b491-01bbe4c8818f_isr_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/23110368-8bbf-40dc-b491-01bbe4c8818f_isr_1124087_20.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 92ae52beddfbcebb9712b88db27b2cbf parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3196024-1417-4243-a66b-d1c651212e28_isr_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3196024-1417-4243-a66b-d1c651212e28_isr_1124087_23.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 22588693ddc89b71be2a039081e4d6a6 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca553a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ad567e8d-30c1-4b03-ba54-f527108cc039_isr_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ad567e8d-30c1-4b03-ba54-f527108cc039_isr_1124087_40.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 3813805de8f8f1d57ae40fb1cda0ab13 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca556c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f16fd25c-b29b-4619-b1ad-8d1da2120f7a_isr_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f16fd25c-b29b-4619-b1ad-8d1da2120f7a_isr_1123958_12.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 07c32a00891bd81b7211772f820f8cf4 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca559e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/0f143f63-a874-4c29-9e56-3ee4c053ef42_isr_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/0f143f63-a874-4c29-9e56-3ee4c053ef42_isr_1123958_13.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 617d824d3406c0473ccf97d796b74f24 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/35d36398-95c8-4b53-8e5e-4923013595ea_isr_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/35d36398-95c8-4b53-8e5e-4923013595ea_isr_1124087_5.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash fcf6a1da334d06a67601935634720893 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3e464a58-173e-4c3d-acd4-08d8b3fbb6a1_isr_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3e464a58-173e-4c3d-acd4-08d8b3fbb6a1_isr_1123958_36.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 863e1870c4eea825784e9a5522dd1206 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9c4bfa0-9c9e-402b-8999-4ec3053e671a_isr_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9c4bfa0-9c9e-402b-8999-4ec3053e671a_isr_1123956_30.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash dcdde0a06a72bf133f77cca8dd5599e5 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/42c12db6-cbfd-4d4b-8384-b1fbdd52e650_isr_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/42c12db6-cbfd-4d4b-8384-b1fbdd52e650_isr_1124087_21.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash f546341c10ed6cdadf122505ffecb30f parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c1925e2b-76a4-4603-a9d2-472a6b3e8072_isr_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c1925e2b-76a4-4603-a9d2-472a6b3e8072_isr_1124087_33.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 198de757e242dccebf51aea79eca571c parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e96c1ccd-522a-4bf0-8cc8-2cad27e4d914_isr_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e96c1ccd-522a-4bf0-8cc8-2cad27e4d914_isr_1123956_37.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 2bfb5560364f3a072bbc3fee8c64c5a1 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/2f3b3363-0375-4008-a158-25a413c46091_isr_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/2f3b3363-0375-4008-a158-25a413c46091_isr_1123954_42.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash d0e4b688a1392b19c00cd7fa486534d0 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca572e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/76727476-967f-49df-93f6-27c5858e81ca_isr_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/76727476-967f-49df-93f6-27c5858e81ca_isr_1123958_49.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 59cd9364dda52a25883bf783c803bd00 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c315e271-8e99-44fd-85d5-1731dec810ca_isr_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c315e271-8e99-44fd-85d5-1731dec810ca_isr_1123958_20.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 3388f3ab7306fc374b037a9800a7f560 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/14d83e19-a4cd-4fc9-8146-06618addd296_isr_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/14d83e19-a4cd-4fc9-8146-06618addd296_isr_1123956_26.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash b8ece6881f492064913f2b9f7bac43b5 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c0063737-b2f6-4bd2-b81e-353b158a0a3e_isr_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c0063737-b2f6-4bd2-b81e-353b158a0a3e_isr_1123958_7.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 544df9b8b61367ee4b24782e0215fb46 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b327b194-d124-4e78-a21d-6e337ad73d26_isr_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b327b194-d124-4e78-a21d-6e337ad73d26_isr_1123958_15.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash ad528cced7f02cc934ec4505a0d5026d parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/fa33193c-780a-40c7-8515-4cfec960a29e_isr_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/fa33193c-780a-40c7-8515-4cfec960a29e_isr_1124087_11.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 28f5e3cb7a3c36a5447f3f79f66c8586 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/829b6cb1-eee3-4530-bd61-0bd0e2795f76_isr_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/829b6cb1-eee3-4530-bd61-0bd0e2795f76_isr_1123954_48.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 810026c723624b0ab134440ab0d1ce61 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9b6afde-4899-432e-9d84-82bda0386f7d_isr_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9b6afde-4899-432e-9d84-82bda0386f7d_isr_1123958_33.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 0fb168998cb2597abdaba91dc1ea499b parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7b02598-1ea9-4477-8580-0402e3390409_isr_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7b02598-1ea9-4477-8580-0402e3390409_isr_1123958_30.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 05f2c881a3b57093b6f4c74e78c83d90 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4817615e-f71a-4a50-a411-5c007f00dea6_isr_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4817615e-f71a-4a50-a411-5c007f00dea6_isr_1123958_5.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash e6204906ff8bb2d3aadc50ed00ba7aa8 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/444f0127-f829-4c08-99fb-7e7291bdb8fe_isr_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/444f0127-f829-4c08-99fb-7e7291bdb8fe_isr_1123956_29.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash a6d21635e6b581ecd35261c4dd3eb0b5 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a759a67b-a71b-4b97-b25a-00062f320c6d_isr_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a759a67b-a71b-4b97-b25a-00062f320c6d_isr_1124087_47.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 7ffdf90bd3712efedb1d830d5cba9b61 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2357d866-f938-4308-916f-07ef828d390a_isr_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2357d866-f938-4308-916f-07ef828d390a_isr_1123956_40.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 2304ca9e7324eb88d90e6216c93c7187 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5a6662e0-7ba1-47b8-9674-011f3424737f_isr_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5a6662e0-7ba1-47b8-9674-011f3424737f_isr_1124087_7.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash d1394271a539e83f1a0ca3435c2d5d5c parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1d96223b-bba7-44a5-897d-8bac30e06c30_isr_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1d96223b-bba7-44a5-897d-8bac30e06c30_isr_1123954_47.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 134f356d411a0c9582b172c36512945c parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/07fa1cc3-6bfd-4b7c-a3aa-e9d732e49d85_isr_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/07fa1cc3-6bfd-4b7c-a3aa-e9d732e49d85_isr_1123956_4.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 8d419e9031b8fd035a022a3f24428fd2 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5433b8da-157d-42b0-b61b-ceb2330403d6_isr_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5433b8da-157d-42b0-b61b-ceb2330403d6_isr_1124087_2.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash f3cfcd9b4912dcd1f2b413270a1b16e3 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b74acd28-6e1a-48aa-aed2-d94a85898bf2_isr_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b74acd28-6e1a-48aa-aed2-d94a85898bf2_isr_1123954_17.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash f29965e14a740c0801332fc923976d74 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/36e1780b-b0f5-4ce9-bc6d-fd99566bca1c_isr_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/36e1780b-b0f5-4ce9-bc6d-fd99566bca1c_isr_1123954_30.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 76165894b28faeb1b59c302f5ef8d3f9 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f69a42c2-9d83-48b8-9683-de57ccc62b8b_isr_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f69a42c2-9d83-48b8-9683-de57ccc62b8b_isr_1123958_56.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash aaaf57570add6f9dadf7c010a1833384 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/059036a3-e5b7-4d62-ab89-76aaefd08e78_isr_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/059036a3-e5b7-4d62-ab89-76aaefd08e78_isr_1123954_3.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash eee6cd5da7519562b3cef88964b96c64 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/12ed3dc4-12a6-4d09-8dec-84d117145150_isr_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/12ed3dc4-12a6-4d09-8dec-84d117145150_isr_1123958_44.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 3077b05e85fa238f2d463489248ed033 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/fc10b91b-c5f2-4ac1-b309-db774121335b_isr_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/fc10b91b-c5f2-4ac1-b309-db774121335b_isr_1123958_3.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 665cc665b7adfb50e0252e023e329db5 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/53dc026f-ee2a-4c19-9d53-8fbad91e50cf_isr_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/53dc026f-ee2a-4c19-9d53-8fbad91e50cf_isr_1123954_46.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash c827e350f07b2edf98dcd89810df9b5c parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/04f303db-d9a2-4dfd-82e7-51bf995199ce_isr_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/04f303db-d9a2-4dfd-82e7-51bf995199ce_isr_1123958_37.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash d20ce38786171aabdda740a8b4b0e5fe parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b73f51a9-6d4c-4bf0-8eec-9074b0e3120e_isr_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b73f51a9-6d4c-4bf0-8eec-9074b0e3120e_isr_1123954_36.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 9996f2bc5348d9b61db7ee03b4062146 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b9f16caf-8eca-46c8-b9e8-413ae3a66352_isr_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b9f16caf-8eca-46c8-b9e8-413ae3a66352_isr_1123956_46.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash be135a1104d723e74b08a454a84dde05 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/5b3e0b97-aeff-4551-ad9c-20b1a1bfbf35_isr_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/5b3e0b97-aeff-4551-ad9c-20b1a1bfbf35_isr_1123956_10.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash e904c47621395765946bb1fb5a94f8dc parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/96edb08d-5ed7-40c3-9918-5edd9450eb9e_isr_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/96edb08d-5ed7-40c3-9918-5edd9450eb9e_isr_1123958_10.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash e9045a10749e913f35c2b59cdb0c7d24 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0f8b126e-303f-4282-8c81-0821b9e805dc_isr_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0f8b126e-303f-4282-8c81-0821b9e805dc_isr_1124087_42.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash c53a788a9c59c82159991ea537fb2d9a parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b8b23447-1655-4c0a-ac8b-480c24d5e186_isr_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b8b23447-1655-4c0a-ac8b-480c24d5e186_isr_1123954_31.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 38a026c4e194eb035c9089f21d07b4fe parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c14cf2a7-d8d2-4bb7-8193-f914f5b80c25_isr_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c14cf2a7-d8d2-4bb7-8193-f914f5b80c25_isr_1123958_25.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 170d457ab7daff7eb94f19a56ca77908 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8c21272b-969e-4903-8ac7-51b3d25d59d6_isr_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8c21272b-969e-4903-8ac7-51b3d25d59d6_isr_1123958_6.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash cffcde1ddfcec7a5c3c5593dc773d46a parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/061e7bfd-b5fe-4e62-9d22-a6690810e704_isr_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/061e7bfd-b5fe-4e62-9d22-a6690810e704_isr_1123958_23.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 1c2761e3ac35230e8976709ec98197a5 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/bb7e261e-09cb-4998-91ad-01c799f3d543_isr_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/bb7e261e-09cb-4998-91ad-01c799f3d543_isr_1123956_25.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash ac2e0e1ff991e3bd5d5e52e5fd1b8153 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e991e8cc-7f76-467b-9603-33d370b7d626_isr_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e991e8cc-7f76-467b-9603-33d370b7d626_isr_1123954_13.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash ce268f63ef3930dce650a2378295ba94 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/11b28325-5d46-4b09-a85d-d849b17899ef_isr_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/11b28325-5d46-4b09-a85d-d849b17899ef_isr_1123956_17.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 9a19c87f92a7add8d3a072124399e602 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/324054db-c529-4156-b612-b30585a9b788_isr_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/324054db-c529-4156-b612-b30585a9b788_isr_1123956_7.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 0b463e0a2821a04cfa3fa97a24a1e72a parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f591a904-2988-4dbc-8811-de243337a42b_isr_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f591a904-2988-4dbc-8811-de243337a42b_isr_1123958_22.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 385ccff954400de553af0954c56a1735 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d19759be-46b4-45f9-bcfb-4ddc0e306a50_isr_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d19759be-46b4-45f9-bcfb-4ddc0e306a50_isr_1123954_19.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 8164237625964d1f0d37b98e2318167f parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1328b0e2-485e-4f73-89f9-734cbbdf36b1_isr_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1328b0e2-485e-4f73-89f9-734cbbdf36b1_isr_1124087_6.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash fad76c6f294cd534fd819c3579d952ab parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1b43bcb3-08f6-4bd6-9ec5-cb13cf7e6b9a_isr_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1b43bcb3-08f6-4bd6-9ec5-cb13cf7e6b9a_isr_1123958_28.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 90b0986a713ffd9a212a134aef97ada6 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9ee6315b-35fb-41d4-898b-7aa8a38cbb91_isr_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9ee6315b-35fb-41d4-898b-7aa8a38cbb91_isr_1123954_50.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash a43d3fe343d4dfd54edaf0e6890e79cc parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9231208-9864-4945-8787-2b52d4a587fd_isr_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9231208-9864-4945-8787-2b52d4a587fd_isr_1123956_45.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash c14c43d53b01a0d45c3c4b1dd3ee42d3 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a416787d-5e1d-467f-a6b5-3c83331f763f_isr_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a416787d-5e1d-467f-a6b5-3c83331f763f_isr_1124087_4.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 502b2706cd419e0ececb23981a326a76 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ede9e5c6-2d9f-491c-b84a-76b2dbfac949_isr_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ede9e5c6-2d9f-491c-b84a-76b2dbfac949_isr_1124087_8.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 9a91342ab620245a7213d61e8266b0b9 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f72e375b-4525-4847-bb05-f487da911ffc_isr_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f72e375b-4525-4847-bb05-f487da911ffc_isr_1124087_39.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 1e8cf7843bb0aa525a0e8ad1e7da1237 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/25dbaed4-cd2f-4827-bfa4-8192b07b4931_isr_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/25dbaed4-cd2f-4827-bfa4-8192b07b4931_isr_1124087_25.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash fcf1e2ef8921b14a4703143280fbbaf3 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/03cf367a-40d8-4486-8263-708b77ffce48_isr_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/03cf367a-40d8-4486-8263-708b77ffce48_isr_1123958_61.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash e4b7767dbf9d036bc8400092a148cc4f parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/44befd7c-93a1-4b6b-9487-993daf04a71e_isr_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/44befd7c-93a1-4b6b-9487-993daf04a71e_isr_1124087_60.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 36775011e4bbe2fbccc401d095c0deaf parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/452ebc2c-2b83-4017-8b54-e2d3c790f9c8_isr_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/452ebc2c-2b83-4017-8b54-e2d3c790f9c8_isr_1123954_51.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 69061c90e1b360980e3432669b1f3e16 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ff7dd953-92b2-48df-a16b-2685c9a8b359_isr_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ff7dd953-92b2-48df-a16b-2685c9a8b359_isr_1123954_41.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 62cd094c54a0ee8622e2ca4122f74d30 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/aadd29df-e339-465a-a4cc-01b9fe70a798_isr_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/aadd29df-e339-465a-a4cc-01b9fe70a798_isr_1124087_16.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash d097b9f09ef83517612e9a5b1ee1d263 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/22cd439c-5cc3-4d94-b0f0-b202f8785ded_isr_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/22cd439c-5cc3-4d94-b0f0-b202f8785ded_isr_1123954_55.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 6bb91d9b15c712b4330a631292faa2eb parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07ff022d-e502-4777-8e19-8c0e0ddfeede_isr_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07ff022d-e502-4777-8e19-8c0e0ddfeede_isr_1124087_24.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 59df167262acc6bb30a6c5e5a1c81724 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8287bd2c-b6bc-49b4-a8e2-1982c4261cd8_isr_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8287bd2c-b6bc-49b4-a8e2-1982c4261cd8_isr_1123958_62.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 356d61e3ba120ff0ae876056a3dab10c parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/705596b8-29f6-4d5c-870f-0d430e04e79e_isr_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/705596b8-29f6-4d5c-870f-0d430e04e79e_isr_1123954_16.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash d043faaf116c552e3b87fe5fc9b2559c parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/75d0e32b-58be-4b60-8af4-b5f46714ff7f_isr_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/75d0e32b-58be-4b60-8af4-b5f46714ff7f_isr_1124087_36.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 0b1bac65126e3219caee2305cedb85b0 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8bc78721-19c7-41ae-a987-1d7f0c3f2513_isr_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8bc78721-19c7-41ae-a987-1d7f0c3f2513_isr_1123956_35.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash b81f1727353ac138923f4b8cef58c38f parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/696d6c20-a2db-4ecb-8eea-7f66f390cc37_isr_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/696d6c20-a2db-4ecb-8eea-7f66f390cc37_isr_1123954_52.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash e52044a6242894bba1152713dab478ee parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/2287ae72-2f8f-48d5-b99c-a437ab82b3fd_isr_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/2287ae72-2f8f-48d5-b99c-a437ab82b3fd_isr_1124087_51.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 941283786a8815fdfcb1a5ddb1bd25bf parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9daa28d-012a-4bca-8b9e-7a97500a542b_isr_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9daa28d-012a-4bca-8b9e-7a97500a542b_isr_1123954_49.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 446d8e561c19c45cfc99cf50d47d7def parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e3ae11d7-d974-4da1-99d6-5d2415e18570_isr_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e3ae11d7-d974-4da1-99d6-5d2415e18570_isr_1123954_9.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash c1f520fc755979a4a2da3d22344ee67e parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8879195d-4af4-4d19-a171-b27ac2a681c7_isr_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8879195d-4af4-4d19-a171-b27ac2a681c7_isr_1123954_33.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 685560191af8fc987295ca63046e6c04 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/6c9f88fe-abd1-4bb8-be36-db595f4e74e8_isr_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/6c9f88fe-abd1-4bb8-be36-db595f4e74e8_isr_1123956_21.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 35ef14e0f004d40cc44f80cd949eeb83 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/28643950-37b0-4a4c-828f-8c88bee6f86a_isr_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/28643950-37b0-4a4c-828f-8c88bee6f86a_isr_1123958_4.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 52fe4915a77bf106488636ca4f555416 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d64d8db7-0d44-4fd4-a16b-6729f4c1d40b_isr_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d64d8db7-0d44-4fd4-a16b-6729f4c1d40b_isr_1123954_18.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash efd4517c5b1b3d2fec272a167e421f6c parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/34b29c6e-028d-4d89-aee6-73f3a39dde98_isr_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/34b29c6e-028d-4d89-aee6-73f3a39dde98_isr_1123954_8.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 59d073453987ec7f14e009b658ba3521 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/52fb89d6-a739-4235-89ed-643373ddd13a_isr_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/52fb89d6-a739-4235-89ed-643373ddd13a_isr_1123958_53.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 59f6224bc8417610fa41c1962460c3d7 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/7460d1dd-f2c2-455b-8117-6a67b8111011_isr_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/7460d1dd-f2c2-455b-8117-6a67b8111011_isr_1124087_30.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 7cee9b4b1c49c73a759d9128446c7d53 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/85643c29-9e15-42ab-8744-2b67a1f7959a_isr_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/85643c29-9e15-42ab-8744-2b67a1f7959a_isr_1123954_62.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 5d3a677eee9225f7b8263c1e31b93a99 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65830161-a482-4857-b1f5-bfbbe154d842_isr_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65830161-a482-4857-b1f5-bfbbe154d842_isr_1123956_57.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 3a59d334562d92dc1e1e5e2ac92d9427 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/da50ffbd-bc45-4ba4-aed1-e6f49b1635c0_isr_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/da50ffbd-bc45-4ba4-aed1-e6f49b1635c0_isr_1124087_14.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash a9728f5406d11f73ded3f8cc4552b241 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/18ef4a72-823b-4c05-8684-1d5e4de86aed_isr_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/18ef4a72-823b-4c05-8684-1d5e4de86aed_isr_1123956_58.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash b65860c8e18dc0f39a3fd31d4c592612 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c12ea14b-803b-4955-905b-25d671200046_isr_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c12ea14b-803b-4955-905b-25d671200046_isr_1123958_51.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 51cd321a906597399b87b63c2c11317a parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c92b88a4-1075-494a-a1c3-1c884d7478aa_isr_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c92b88a4-1075-494a-a1c3-1c884d7478aa_isr_1123956_14.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 68f81449717d3ec91b189f2a36d7c25d parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c0fa8a0-2e25-4f86-934a-401be3d32ef7_isr_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c0fa8a0-2e25-4f86-934a-401be3d32ef7_isr_1123958_39.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 8b84f33d8c0b7924829b850eb896a7af parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/47732bfb-e82b-4b22-9bc0-f3d14903c229_isr_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/47732bfb-e82b-4b22-9bc0-f3d14903c229_isr_1123954_26.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash d162e79be483012897b0761f218d6f23 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0b85f343-a708-4f8b-949d-f0ce503f0ef1_isr_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0b85f343-a708-4f8b-949d-f0ce503f0ef1_isr_1124087_50.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 9e3f7bda762fb4076adf4bc0f0bb5001 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/517bb18e-e3e2-4f2e-9c3c-0f127a67b1c9_isr_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/517bb18e-e3e2-4f2e-9c3c-0f127a67b1c9_isr_1123958_31.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 4af972ee46c663e55eb4b00132fa6a8e parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/957150f1-f4ea-4c12-959a-2920ce07da99_isr_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/957150f1-f4ea-4c12-959a-2920ce07da99_isr_1123956_15.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 630b0ceecc2eee142bafe9366a8344da parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f91a55c-d08e-45b6-802e-2925ae0e0593_isr_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f91a55c-d08e-45b6-802e-2925ae0e0593_isr_1123954_40.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 8f338e66fc30e748ff9a777ab2b56d28 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a89be1a6-681f-4da9-b6f3-5952a92855eb_isr_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a89be1a6-681f-4da9-b6f3-5952a92855eb_isr_1124087_56.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash af7ba658275e9792722a2c31aa53a20e parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3d6c6545-174b-4113-934a-876f4dea32ca_isr_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3d6c6545-174b-4113-934a-876f4dea32ca_isr_1124087_32.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash e496c40d2e34532e35b71ba66a8d45b0 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9c83036-a1d3-4523-bb31-4bc49fa1ba23_isr_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9c83036-a1d3-4523-bb31-4bc49fa1ba23_isr_1123954_1.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash ff7c7aa561d651368dd6b1380d7a4a8e parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a811ac9-f3aa-4151-9ac7-b011984238f6_isr_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a811ac9-f3aa-4151-9ac7-b011984238f6_isr_1124087_27.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash f3e9d792fd2576479e6e12975159810a parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/4ec7e3d9-bceb-4ada-9459-1545f5fbaca3_isr_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/4ec7e3d9-bceb-4ada-9459-1545f5fbaca3_isr_1123956_52.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 62208f1ede0b3a22267e8a6c24051089 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d84f8387-8865-4ca9-bc2c-88b1f6be760c_isr_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d84f8387-8865-4ca9-bc2c-88b1f6be760c_isr_1123954_7.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 75f8136c5049ef586daa72a99d5be832 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/228cb8fd-0757-49e8-81a5-b0306688f448_isr_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/228cb8fd-0757-49e8-81a5-b0306688f448_isr_1123956_11.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 1159e99d8163a181c7ee11be882ee83d parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4aac3d9-b30f-453f-b74d-c5cfa7b51164_isr_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4aac3d9-b30f-453f-b74d-c5cfa7b51164_isr_1123958_27.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 3f10e575bb39d51d2c0228e453ca0d3e parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/fff04bc1-131c-499d-9d66-faebdfe0066f_isr_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/fff04bc1-131c-499d-9d66-faebdfe0066f_isr_1123956_31.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 97939bbcfc3da45ef0d0af776c32d0f5 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8aea0967-8c1e-41e0-bb1e-696fd37cdc73_isr_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8aea0967-8c1e-41e0-bb1e-696fd37cdc73_isr_1124087_35.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 59d3e71db3b62e9db5490ef362ff27d3 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/309472f6-0dfa-407c-8db5-bd927a1587b5_isr_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/309472f6-0dfa-407c-8db5-bd927a1587b5_isr_1123958_21.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash b2f58ed078c57c2d295637aa4adb18c8 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/608be376-8e61-4b62-9f1e-292a4444287b_isr_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/608be376-8e61-4b62-9f1e-292a4444287b_isr_1124087_38.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 13fccdffc99363e0ebc6c78553ab0131 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3e1e743-f182-45aa-b0a3-626a12936e18_isr_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3e1e743-f182-45aa-b0a3-626a12936e18_isr_1124087_41.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 7baf958b49b5fce7dfafff386a071cb9 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/db76f1b9-9cae-468c-a16f-be179ab96712_isr_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/db76f1b9-9cae-468c-a16f-be179ab96712_isr_1123954_11.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 76de56a32d90506c0182e4ef5bae25d4 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/e6ef47fa-6786-4edb-aaa5-0acbf5e95499_isr_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/e6ef47fa-6786-4edb-aaa5-0acbf5e95499_isr_1124087_52.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 15ffc855f0c5bdf6784226655149b4f3 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f4f1569-2407-4f4c-960f-9c138b9d8139_isr_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f4f1569-2407-4f4c-960f-9c138b9d8139_isr_1123954_56.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 3bd77d0922baba32f736b01bb64b6cfb parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1b4db1c8-e075-4771-a702-dabc1a1a382c_isr_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1b4db1c8-e075-4771-a702-dabc1a1a382c_isr_1124087_46.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 7359c7abdf54dfb7760cf707528c1cb7 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 03:37:26 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a2ae7be4-7687-423d-b1ee-a3eba746b1c1_isr_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a2ae7be4-7687-423d-b1ee-a3eba746b1c1_isr_1123956_22.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 7b0c7a11ff76d8b41910312f8115cf9b parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/0b7b76a9-a74b-4a46-952e-7fde7345c9a3_isr_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/0b7b76a9-a74b-4a46-952e-7fde7345c9a3_isr_1123954_34.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 9d550bb9224029f4b48f63afa47710cc parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bc65d877-52c4-4a05-83a0-3ad7b188eb91_isr_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bc65d877-52c4-4a05-83a0-3ad7b188eb91_isr_1124087_1.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash e1a4c6ec18d6537d42c6a88b16912a07 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3c1bcd56-7d78-49f4-96cd-4ba3ac09f44c_isr_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3c1bcd56-7d78-49f4-96cd-4ba3ac09f44c_isr_1123956_60.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash dfe6cd75394320a84cb6061654d4235f parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/037ac91f-0bb8-4af8-8c42-582203dd0c3e_isr_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/037ac91f-0bb8-4af8-8c42-582203dd0c3e_isr_1123954_54.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 1b4b454a930106673dad01262e2c7893 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 132 parsl.dataflow.strategy DEBUG: Executor multi has 132 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 32 more blocks parsl.executors.status_handling INFO: Scaling out by 32 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 03:37:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 03:37:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:37:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/283b1c8b-bf1d-4445-b9f8-1f915fd389cf_isr_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/283b1c8b-bf1d-4445-b9f8-1f915fd389cf_isr_1123956_5.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash f390a1deb14ab211e74e7eab721686e3 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e0ce240e-0758-47d6-938e-bfb17102851f_isr_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e0ce240e-0758-47d6-938e-bfb17102851f_isr_1123956_44.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 9f7ba0e5a66fbbb4781dcd6bfd46c734 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfda0d70-d81e-4310-b263-693c0b1dfb57_isr_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfda0d70-d81e-4310-b263-693c0b1dfb57_isr_1123954_23.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 16e53b055975f3a48fa3674e5592a35c parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b795e720-acbc-440a-a558-b9573ebf638a_isr_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b795e720-acbc-440a-a558-b9573ebf638a_isr_1123954_29.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 83b810c3c05a47eefa59e2b55a42eef4 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b67b6e0d-6c52-485e-8206-d781b12478ee_isr_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b67b6e0d-6c52-485e-8206-d781b12478ee_isr_1123956_38.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 6539ae7d5bfc40e5255062eb4f2829ee parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/9cd616a1-699c-4210-8046-037992974b74_isr_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/9cd616a1-699c-4210-8046-037992974b74_isr_1124087_55.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 50a55e181fa574ef523efa6eba4d82c9 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8b69b961-27e7-4f30-9b0c-9ae9f5e4627f_isr_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8b69b961-27e7-4f30-9b0c-9ae9f5e4627f_isr_1124087_29.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 2f5677d106d5a91e253fd84452d2335b parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a1d92bdd-13a8-4e86-b557-9f609e3c27c8_isr_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a1d92bdd-13a8-4e86-b557-9f609e3c27c8_isr_1124087_28.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 6981d26f36ac595f15bda165d279131b parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1da25323-a804-4f44-b102-016d96bfc1fd_isr_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1da25323-a804-4f44-b102-016d96bfc1fd_isr_1123956_41.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash cf80dcc85132e64406825b17c0e2cca0 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/db2535a6-119d-4326-82c3-248504abd2bf_isr_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/db2535a6-119d-4326-82c3-248504abd2bf_isr_1123958_54.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash cba48ade0d90681e53b970b3561c3ba2 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e5182262-280b-46c4-9b60-aa1ef6cba5d9_isr_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e5182262-280b-46c4-9b60-aa1ef6cba5d9_isr_1123954_43.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash e4f5e4f6968e7ec9df02ff8c49163015 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1298a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9490ad4-55a3-4b20-a96a-cc0efcdf730c_isr_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9490ad4-55a3-4b20-a96a-cc0efcdf730c_isr_1123958_9.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 28dcb4916e08637961d65da073fa5775 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ae34244f-9ca5-4244-a09a-bf8a9b01f1f0_isr_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ae34244f-9ca5-4244-a09a-bf8a9b01f1f0_isr_1123954_60.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash af1c5229cbdc342cc8285c3346401db9 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1291c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a3d4f5b1-b01c-4a74-89cd-f77a5978f078_isr_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a3d4f5b1-b01c-4a74-89cd-f77a5978f078_isr_1123956_19.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash bbbc5ce51646d4a2884dcd68c7509450 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/987a884c-3dfa-48c4-a94b-7855b1df7611_isr_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/987a884c-3dfa-48c4-a94b-7855b1df7611_isr_1123954_21.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 82635080c0ce9bb72b90e6825e692344 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/315aca90-503e-4827-a75e-d0f68662ab15_isr_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/315aca90-503e-4827-a75e-d0f68662ab15_isr_1123958_26.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 52ba6f1c719b91263bfbeda1343968d0 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f4794909-308b-4d16-af1d-b5c40ddc9b6e_isr_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f4794909-308b-4d16-af1d-b5c40ddc9b6e_isr_1124087_13.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash be39d3cd8653f74bd8927b5d9f75678f parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8df46b03-615f-407a-8f0b-ffd4b29e586f_isr_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8df46b03-615f-407a-8f0b-ffd4b29e586f_isr_1123958_48.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash c10e43bda3a9e943b199038fa8dbe5cd parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3bc21b3b-6267-43d5-b0e1-3761e38465b3_isr_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3bc21b3b-6267-43d5-b0e1-3761e38465b3_isr_1123958_8.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 789618aa73641cf9ac78b90a57b74d33 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1c092786-b9d7-4cb5-88e3-b0dae5b27ebe_isr_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1c092786-b9d7-4cb5-88e3-b0dae5b27ebe_isr_1123956_8.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 68feb349f3c746714113d5d5570684ee parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7dda3b50-e575-49ec-8f1d-1b29e5991d04_isr_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7dda3b50-e575-49ec-8f1d-1b29e5991d04_isr_1123956_12.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 94da4bc718d0188f9adb0c909e7c8a00 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/73ac543a-9d03-4c0f-a9c0-66ffce927bf1_isr_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/73ac543a-9d03-4c0f-a9c0-66ffce927bf1_isr_1123958_42.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash fa0ab7d9a92783bac2efcddd35c7d5d7 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9f27e7cc-f627-468f-b358-90af8c6b17ed_isr_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9f27e7cc-f627-468f-b358-90af8c6b17ed_isr_1123958_24.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 682823ada065ff011d04966679b9005f parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07f1f4f9-08c2-498e-acb4-a5f4e8275023_isr_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07f1f4f9-08c2-498e-acb4-a5f4e8275023_isr_1124087_57.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 7d05e6b29db844e9f43e9be62eb2e97a parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 03:37:26 proc_lsst.multi:170 [INFO] [multi] job_id 16884260 proc_lsst.multi INFO: [multi] job_id 16884260 2024-03-07 03:37:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884260 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884260 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16884260 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 03:37:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 03:37:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/d154e555-4eda-41e6-961c-a37f3b869d34_isr_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/d154e555-4eda-41e6-961c-a37f3b869d34_isr_1123956_24.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash c1776ae8b3dd1f3459acf3182b8aab6e parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/d3e7a4e8-d9aa-4a5a-a926-01d444fd89b7_isr_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/d3e7a4e8-d9aa-4a5a-a926-01d444fd89b7_isr_1124087_12.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 8f0c1df8bd762c9c2aa3d47d5c801b86 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/3a082f0f-c861-4712-a38d-4585eae9d424_isr_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/3a082f0f-c861-4712-a38d-4585eae9d424_isr_1123954_38.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 956d6170d7864e40529f2cf3f7eda9ef parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dcf25a6f-a6cb-4ed2-8e12-9abaf0d5ae04_isr_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dcf25a6f-a6cb-4ed2-8e12-9abaf0d5ae04_isr_1123954_27.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 77374421f86fb6ab6f8827eed3f346d5 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3753fa64-a50e-4963-bdbc-b7defef899c1_isr_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3753fa64-a50e-4963-bdbc-b7defef899c1_isr_1123956_18.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 1aef54c419e4566206aefe4ed56da7f9 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/97db91c3-6edd-43ed-8066-9de5e6806d5f_isr_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/97db91c3-6edd-43ed-8066-9de5e6806d5f_isr_1123956_59.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 64ce5613223dfc83dd8b14e39c827017 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/6c48e18d-7626-4877-a962-e6fda789091e_isr_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/6c48e18d-7626-4877-a962-e6fda789091e_isr_1124087_17.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash f936eb3caf2be492db486bf39e0d4d81 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1ff549b3-c59b-4a6d-a203-4ded6e8a8997_isr_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1ff549b3-c59b-4a6d-a203-4ded6e8a8997_isr_1124087_48.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 5ee132f759887c2b34304931013d462e parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873240e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/f3070a0b-b96a-4f0d-bfaf-b68a7a11046d_isr_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/f3070a0b-b96a-4f0d-bfaf-b68a7a11046d_isr_1123956_1.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 96935d6dd696e61adf86dd966e9bf386 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/924fb9f6-c471-4f09-87bb-40135b5204bd_isr_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/924fb9f6-c471-4f09-87bb-40135b5204bd_isr_1123954_25.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash c3ae1cc3f994746d98e08693bebf5012 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/25365f10-e760-4c94-aa21-876bf56cb88b_isr_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/25365f10-e760-4c94-aa21-876bf56cb88b_isr_1123954_45.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 9e7584268b0ef0f2cca5537355bb35bb parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98c1821c-3d46-40fa-b1c4-e8b7a60f7d3f_isr_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98c1821c-3d46-40fa-b1c4-e8b7a60f7d3f_isr_1123954_15.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash b9800e24b34638fe7518f8fd682793cc parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfce2375-9a62-41c0-8d9b-75b89acff708_isr_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfce2375-9a62-41c0-8d9b-75b89acff708_isr_1123954_53.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash b6b5230c1faa1e1429da480a567fc8f1 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dffc42f3-8ca7-46f5-959c-5304f5b29560_isr_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dffc42f3-8ca7-46f5-959c-5304f5b29560_isr_1123954_5.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 921a42e398c86bbc77425a5da034c28a parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/34afe697-0f70-4a66-9cc2-945b667bc208_isr_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/34afe697-0f70-4a66-9cc2-945b667bc208_isr_1123958_43.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 9863495f1c84aa1fe8828ce666cace44 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1723e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c4352610-e962-48b7-b464-305eac7c1775_isr_1123956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c4352610-e962-48b7-b464-305eac7c1775_isr_1123956_36.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash d666fa808594046936a83f89fc2edf81 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/13fe12a9-2f42-44de-9d39-617b12595670_isr_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/13fe12a9-2f42-44de-9d39-617b12595670_isr_1123956_51.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 70a1766168f7634c84a744ccd8019227 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bb4c774c-1d28-4b66-b9f2-6926cd5b8a07_isr_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bb4c774c-1d28-4b66-b9f2-6926cd5b8a07_isr_1123954_4.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash d42e458f33444690e2fbddefab8958d9 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/721d29eb-3cf7-4a0c-b081-1bba25bb0cec_isr_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/721d29eb-3cf7-4a0c-b081-1bba25bb0cec_isr_1123954_39.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 3916f14471552e4f4062e7b50afff00a parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a3b3c7e7-d299-4dc0-8862-1b2be7016f44_isr_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a3b3c7e7-d299-4dc0-8862-1b2be7016f44_isr_1123954_57.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash bc88ab4dd86c2534d609161cd490e56d parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b33012aa-aa26-4c3f-98f8-141afadb4352_isr_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b33012aa-aa26-4c3f-98f8-141afadb4352_isr_1124087_58.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 89b48f79130b8df724d256e5d4482c9c parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3e80bdd1-c28b-4e3e-930c-1c376a984388_isr_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3e80bdd1-c28b-4e3e-930c-1c376a984388_isr_1124087_61.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 207bf81c68cfa3e45e8c491b26ac13a4 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2feb6b4f-f0c9-44c2-b9b0-348c5b171774_isr_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2feb6b4f-f0c9-44c2-b9b0-348c5b171774_isr_1123956_49.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash e8bd283e18e237adfa95e433c5e544c0 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ce1e0181-b065-488a-bd24-5c8e5f0c410e_isr_1123956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ce1e0181-b065-488a-bd24-5c8e5f0c410e_isr_1123956_55.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash f915088e6ca7c9337815b8d4e3b2f0f2 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873253a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/915ba7b6-04e8-4443-90fa-740470cd6b60_isr_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/915ba7b6-04e8-4443-90fa-740470cd6b60_isr_1124087_62.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash b3768843225fa2cc7cf05002ec927c36 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873256c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e795a5fe-63f2-4182-8aa5-d990dd741963_isr_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e795a5fe-63f2-4182-8aa5-d990dd741963_isr_1123958_52.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 019307c3a3dfe7724e5430d4a825af1f parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873259e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8b7c4646-136a-4771-a82d-abb51f207201_isr_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8b7c4646-136a-4771-a82d-abb51f207201_isr_1123958_14.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 49c336dc39848eb4170d91c69017d56d parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/f3a8d742-8069-4e0b-801f-b7673f18d51d_isr_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/f3a8d742-8069-4e0b-801f-b7673f18d51d_isr_1123954_20.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 8709b6193a61497ae82c3c31f337bc03 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec2f818a-4a34-4add-98d4-e93321450e92_isr_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec2f818a-4a34-4add-98d4-e93321450e92_isr_1123956_20.stderr 2024-03-07 03:37:27 proc_lsst.multi:170 [INFO] [multi] job_id 16884261 proc_lsst.multi INFO: [multi] job_id 16884261 2024-03-07 03:37:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884261 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884261 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16884261 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 03:37:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 03:37:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash fd122ecb55ac075b6575e3570e89c423 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e4fc7132-a22b-429d-8245-f49b5c476098_isr_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e4fc7132-a22b-429d-8245-f49b5c476098_isr_1123954_58.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 4d21d0b667af227fbc67812208f34a81 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a777548e-70a0-4a9d-bdc7-3846f50e28b9_isr_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a777548e-70a0-4a9d-bdc7-3846f50e28b9_isr_1123958_50.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash f7c77b9bcde0e1cef2d9c8001ba89883 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3a0a9e2d-7da7-4539-b1ad-c0b77cd0b3f9_isr_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3a0a9e2d-7da7-4539-b1ad-c0b77cd0b3f9_isr_1124087_15.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 21928088fb0f26fe3c464105c226bb6d parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/75861e7f-4dd2-454a-a65f-b9f72228e364_isr_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/75861e7f-4dd2-454a-a65f-b9f72228e364_isr_1123958_55.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash ce61961ee50c1e8d7e78058e8e1c8e96 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8da616e1-725f-43af-b32e-9e57e19c0838_isr_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8da616e1-725f-43af-b32e-9e57e19c0838_isr_1123956_39.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 610b4bb6b9239c22567c0f655f96bafc parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/85cc6439-692e-43a5-877b-e834843fe89c_isr_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/85cc6439-692e-43a5-877b-e834843fe89c_isr_1123956_16.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash a6c52bfe816d7ed7df71b8446aa87e3f parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b06d124b-bc58-46f0-8c0a-f05d0a17364c_isr_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b06d124b-bc58-46f0-8c0a-f05d0a17364c_isr_1124087_44.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 2dbe16b3332fed47c42a95eed09e40ed parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/17433f0a-fbd6-450f-850f-05c735643d89_isr_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/17433f0a-fbd6-450f-850f-05c735643d89_isr_1124087_10.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 1ed8e98bf6c53b7175f0d41b63c8f2a8 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873276a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/083acf69-2a2a-424f-a8a6-db438c28cb9d_isr_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/083acf69-2a2a-424f-a8a6-db438c28cb9d_isr_1123958_19.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 35f35a7b3cea950836108d2785601086 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873279c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1bed2145-3b2e-4a4c-9082-17a2709f83c9_isr_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1bed2145-3b2e-4a4c-9082-17a2709f83c9_isr_1123954_28.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 5022068ddaa9eae2294b7c81ab1dcd00 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dc979ffe-ec1e-489b-b6d6-a486cba6212c_isr_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dc979ffe-ec1e-489b-b6d6-a486cba6212c_isr_1123956_61.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 2418809a2400448e3ff7a5ee7e5aaa5b parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e1be5409-c81d-4160-b560-fad915cc0b04_isr_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e1be5409-c81d-4160-b560-fad915cc0b04_isr_1123954_2.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash a3eb5ed2b4d0741cb30443cbd089b65f parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b671a302-f1f7-4d2e-a0f9-58d4c7678ca2_isr_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b671a302-f1f7-4d2e-a0f9-58d4c7678ca2_isr_1123954_24.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash c317a812bba5d2990cdae92f253a4c09 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98a6636a-20b6-4a94-8417-dccdd3e7497b_isr_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98a6636a-20b6-4a94-8417-dccdd3e7497b_isr_1123954_35.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash c2b106a3534b725afcc3edce088c83cb parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/266c88b2-1b04-4047-be8f-e747dd171401_isr_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/266c88b2-1b04-4047-be8f-e747dd171401_isr_1124087_45.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash e5d51dc82498b2afdc42613f32d27359 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ae4ad2c5-24a1-4648-b669-850fc8a673f5_isr_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ae4ad2c5-24a1-4648-b669-850fc8a673f5_isr_1123956_28.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 109347692e688335e3bddaf22bc02bcc parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/33c39191-92f2-476e-963a-d2784666aaf3_isr_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/33c39191-92f2-476e-963a-d2784666aaf3_isr_1123956_53.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 884df25ba0385bc5d0141d8c7e83a1ab parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4ff7e57-87ed-41f2-bfae-d47cd6713ea6_isr_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4ff7e57-87ed-41f2-bfae-d47cd6713ea6_isr_1123958_60.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 30d4a745d6972c89d176f8cea328f64b parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8a1c379e-9d9e-41bc-90d9-4a2089a4007e_isr_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8a1c379e-9d9e-41bc-90d9-4a2089a4007e_isr_1123954_22.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash acc58f1b625748ee7b8b73a181003ad8 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a7f8db4-0bc2-4a7d-b80e-870a03f82aa4_isr_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a7f8db4-0bc2-4a7d-b80e-870a03f82aa4_isr_1124087_49.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 697d7df3bbc0f40e7d8f8eb28f5607bb parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b8d14d61-efef-4aaf-9859-7b9eeee838f7_isr_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b8d14d61-efef-4aaf-9859-7b9eeee838f7_isr_1123958_38.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash ade245d754136e38253501982c8bc560 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/717bde92-c332-4576-a4b0-5ab4d408cad9_isr_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/717bde92-c332-4576-a4b0-5ab4d408cad9_isr_1123956_50.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 7be4b55c943168027b32aaf9e8cd2556 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6fe81655-9f68-4835-adc4-93a3a7fcace8_isr_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6fe81655-9f68-4835-adc4-93a3a7fcace8_isr_1123954_44.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping 2024-03-07 03:37:27 proc_lsst.multi:170 [INFO] [multi] job_id 16884262 proc_lsst.multi INFO: [multi] job_id 16884262 2024-03-07 03:37:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884262 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884262 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16884262 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 03:37:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 03:37:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 59a150689cd108c0cab67c744de149e4 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c37daf8c-0590-48f9-8233-329770484eb8_isr_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c37daf8c-0590-48f9-8233-329770484eb8_isr_1124087_31.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash d67addd86c6797badf1f16072b7b7b82 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b845de0c-0aa1-4d06-98be-5bb63e430e64_isr_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b845de0c-0aa1-4d06-98be-5bb63e430e64_isr_1124087_59.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash f16d664b17fa9a475afb03896adabbaa parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b7a4f39c-cd5f-47f2-85b8-0f7d7187c709_isr_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b7a4f39c-cd5f-47f2-85b8-0f7d7187c709_isr_1123956_48.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash d48a5a24c75bb73596c2a3d154653be1 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/598291ed-5525-40e0-8389-694a9b684fbc_isr_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/598291ed-5525-40e0-8389-694a9b684fbc_isr_1123954_10.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash d012051f473313613c22e4a2857a0677 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/6e85d617-0995-48ea-990c-3482d7d8576d_isr_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/6e85d617-0995-48ea-990c-3482d7d8576d_isr_1123958_16.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 6bc8e7f3cae018a31ce958f5356cda1a parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/165a30ea-056f-4d38-ae0e-d7a592f3151d_isr_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/165a30ea-056f-4d38-ae0e-d7a592f3151d_isr_1124087_9.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 63765065ef40ab496509fd5a7a999c84 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a263dd41-9085-4a08-8bc5-19a3aefb4e7c_isr_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a263dd41-9085-4a08-8bc5-19a3aefb4e7c_isr_1123954_12.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 41b4a59ee659ee8fba4570b03bbcc810 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a0b08c4c-2fac-49fe-bd64-ecd03ba08ecf_isr_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a0b08c4c-2fac-49fe-bd64-ecd03ba08ecf_isr_1123956_43.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 8f8ac01b6e777c4a3c8c45ab358c4014 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/99356537-0e99-47ac-bdc7-77335393996a_isr_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/99356537-0e99-47ac-bdc7-77335393996a_isr_1123956_6.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 671253f2db013bdf26ec78b29fc1cd8c parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/300d0f50-48fa-4af3-a143-29fc5d09f74f_isr_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/300d0f50-48fa-4af3-a143-29fc5d09f74f_isr_1123954_6.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 90821d35a30c5d92c6cb2baf77b2b237 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5abb4c45-e01b-46b4-8a41-e926aa9a814c_isr_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5abb4c45-e01b-46b4-8a41-e926aa9a814c_isr_1124087_22.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash 9c77fa41ab1764794277bbb394fe21bd parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec0478e6-d38e-426f-82cb-2f98f9f70352_isr_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec0478e6-d38e-426f-82cb-2f98f9f70352_isr_1123956_56.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash f68fa866a529562417c936d934be3021 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/19ecc46a-0bc9-4fb4-b971-493839d1595d_isr_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/19ecc46a-0bc9-4fb4-b971-493839d1595d_isr_1123956_47.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 97d41bd2d351c9cc6bba31cf9d0e3477 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9360a82b-58e0-4fdd-a4eb-0dfea60807ff_isr_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9360a82b-58e0-4fdd-a4eb-0dfea60807ff_isr_1123958_41.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 00f6a65d5e21a7b760f7270e17c82b33 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7564522-09be-489d-b8b7-d2681b1b31ec_isr_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7564522-09be-489d-b8b7-d2681b1b31ec_isr_1123958_32.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash a58ed0a53fbd72854a6bb72d46a49fbd parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/84e04b2b-00a3-4f55-b368-f771c7d2fc4c_isr_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/84e04b2b-00a3-4f55-b368-f771c7d2fc4c_isr_1123956_9.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash f661dea83cd43627b57da1aed24fdc39 parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/33fd80e5-432c-454d-9154-fe5db47f8ca6_isr_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/33fd80e5-432c-454d-9154-fe5db47f8ca6_isr_1123958_29.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 36c508e4b86d4bb672407c6976a4726a parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e5877ca5-7337-4ea6-a52d-952ea7d079b8_isr_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e5877ca5-7337-4ea6-a52d-952ea7d079b8_isr_1123956_42.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash bd516d99cc07b2b5f268c14f5bc4c591 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/590c566e-aeb6-44c4-a18e-a8c135e160f1_isr_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/590c566e-aeb6-44c4-a18e-a8c135e160f1_isr_1123956_23.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 8bd238e42a70a5f7b39f3cbd4212f702 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/02b96a78-dbca-4b25-a31a-49cd0ff6a606_isr_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/02b96a78-dbca-4b25-a31a-49cd0ff6a606_isr_1124087_26.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash ece3d4d0554d5470ff3eb4eb8f3626e7 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/40f2305f-316a-4745-adce-701d571f6508_isr_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/40f2305f-316a-4745-adce-701d571f6508_isr_1123956_33.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 34bb5b09cadb2c43eb829c4f150c7bbc parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3fca6a36-a2c3-41db-99fe-13587a4b605f_isr_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3fca6a36-a2c3-41db-99fe-13587a4b605f_isr_1123958_2.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 16037f19d7fa742f2945132de45a0354 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4d6dab0f-6191-43e2-8192-4b725418c82e_isr_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4d6dab0f-6191-43e2-8192-4b725418c82e_isr_1123958_40.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash c2e9c9a813c0ddf2e12de2aecdd4db5f parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7772a733-4f14-479b-9cb4-123cbc96b262_isr_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7772a733-4f14-479b-9cb4-123cbc96b262_isr_1123956_62.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 8b3d289480f5ebd6a6f8ad398cde2c7a parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/82812d42-4be7-4fdb-8f31-ceebbfae4060_isr_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/82812d42-4be7-4fdb-8f31-ceebbfae4060_isr_1124087_19.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 859f3acff31733d7815ed6940b930fdd parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/16e224d3-1269-42e5-8919-2e6304c06832_isr_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/16e224d3-1269-42e5-8919-2e6304c06832_isr_1123958_46.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash f511da540448cf7812a0925cc32fd6e1 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/830f99cd-a19d-4ec8-858e-128efc8a291f_isr_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/830f99cd-a19d-4ec8-858e-128efc8a291f_isr_1123954_37.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash c40756aebd6f0505e68a24ac3b302424 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a36aaed2-9357-4abf-ab47-2f4d0ce87966_isr_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a36aaed2-9357-4abf-ab47-2f4d0ce87966_isr_1123954_14.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 96f09fd278915d06e77aeeb1d1b82b95 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b0f841c1-5ea4-43d8-84fe-b78c307340ab_isr_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b0f841c1-5ea4-43d8-84fe-b78c307340ab_isr_1123956_34.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 6e3d1d3d88b56e26933e862200513f2b parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/0ac1a99b-94a8-4c80-900a-8d775c98eb9b_isr_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/0ac1a99b-94a8-4c80-900a-8d775c98eb9b_isr_1123956_13.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 316c4fc13742d2091d43ce995cccf716 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468872189a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9b26dc33-e841-45e5-a793-efd2dfa10855_isr_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9b26dc33-e841-45e5-a793-efd2dfa10855_isr_1123954_59.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 82741a32be1f54483047d5758a9f419d parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8941d282-774c-425f-b0e1-ae15b3964492_isr_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8941d282-774c-425f-b0e1-ae15b3964492_isr_1123958_59.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 3588fbc93366f6b9ccbfd9827a47cf05 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1323fb68-b2b6-4987-ba62-cd9a2b848377_isr_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1323fb68-b2b6-4987-ba62-cd9a2b848377_isr_1123958_35.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash db86ad77e0cbbbde5654cf9276b0c9ef parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c7991d7-2bc2-4871-ae52-1f6757096c1c_isr_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c7991d7-2bc2-4871-ae52-1f6757096c1c_isr_1123958_47.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash a994484d4ac8e269a13615252820f100 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/900ce0db-501b-40b6-a2cd-a1b7c7ced1ef_isr_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/900ce0db-501b-40b6-a2cd-a1b7c7ced1ef_isr_1123958_18.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash ef94093cb4853c7b408e7123e6f662d4 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/08df6f8c-d953-43d5-b1c7-ab42dcce212c_isr_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/08df6f8c-d953-43d5-b1c7-ab42dcce212c_isr_1123954_32.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash fb919814f5d8003d9334c1ddc42c51c8 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/cdc98367-e89d-4f61-a6b0-639bfeec7c9e_isr_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/cdc98367-e89d-4f61-a6b0-639bfeec7c9e_isr_1124087_34.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 4216aabc98c21d5b53c3c08824c54b18 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8c9ea65b-7ed8-43f9-92da-af52325ef001_isr_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8c9ea65b-7ed8-43f9-92da-af52325ef001_isr_1123956_54.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 366b16a6f534092e9c0ca74a00ed1914 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688721a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8ed28c84-1a05-40ec-8c15-0861ad02e06c_isr_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8ed28c84-1a05-40ec-8c15-0861ad02e06c_isr_1123958_34.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash ef8028e74bc49a63321d7791f0ba815e parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688721a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/777917ee-ad9a-43b3-8709-36faa9375245_isr_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/777917ee-ad9a-43b3-8709-36faa9375245_isr_1124087_53.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping 2024-03-07 03:37:28 proc_lsst.multi:170 [INFO] [multi] job_id 16884263 proc_lsst.multi INFO: [multi] job_id 16884263 2024-03-07 03:37:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884263 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884263 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16884263 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 03:37:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 03:37:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:28 proc_lsst.multi:170 [INFO] [multi] job_id 16884264 proc_lsst.multi INFO: [multi] job_id 16884264 2024-03-07 03:37:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884264 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884264 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16884264 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 03:37:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 03:37:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:28 proc_lsst.multi:170 [INFO] [multi] job_id 16884265 proc_lsst.multi INFO: [multi] job_id 16884265 2024-03-07 03:37:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:37:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884265 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884265 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16884265 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 03:37:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 03:37:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:37:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:29 proc_lsst.multi:170 [INFO] [multi] job_id 16884266 proc_lsst.multi INFO: [multi] job_id 16884266 2024-03-07 03:37:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:37:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884266 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884266 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16884266 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 03:37:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:29 proc_lsst.multi:170 [INFO] [multi] job_id 16884267 proc_lsst.multi INFO: [multi] job_id 16884267 2024-03-07 03:37:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:37:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884267 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884267 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16884267 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 03:37:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:29 proc_lsst.multi:170 [INFO] [multi] job_id 16884268 proc_lsst.multi INFO: [multi] job_id 16884268 2024-03-07 03:37:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:37:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884268 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884268 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16884268 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 03:37:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:29 proc_lsst.multi:170 [INFO] [multi] job_id 16884269 proc_lsst.multi INFO: [multi] job_id 16884269 2024-03-07 03:37:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:37:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884269 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884269 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16884269 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 03:37:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:37:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:30 proc_lsst.multi:170 [INFO] [multi] job_id 16884270 proc_lsst.multi INFO: [multi] job_id 16884270 2024-03-07 03:37:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:37:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884270 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884270 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16884270 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 03:37:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:30 proc_lsst.multi:170 [INFO] [multi] job_id 16884271 proc_lsst.multi INFO: [multi] job_id 16884271 2024-03-07 03:37:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:37:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884271 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884271 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16884271 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 03:37:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:30 proc_lsst.multi:170 [INFO] [multi] job_id 16884272 proc_lsst.multi INFO: [multi] job_id 16884272 2024-03-07 03:37:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:37:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884272 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884272 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16884272 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 03:37:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:30 proc_lsst.multi:170 [INFO] [multi] job_id 16884273 proc_lsst.multi INFO: [multi] job_id 16884273 2024-03-07 03:37:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:37:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884273 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884273 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16884273 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 03:37:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:37:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:31 proc_lsst.multi:170 [INFO] [multi] job_id 16884274 proc_lsst.multi INFO: [multi] job_id 16884274 2024-03-07 03:37:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:37:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884274 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884274 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16884274 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 03:37:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 03:37:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:37:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:31 proc_lsst.multi:170 [INFO] [multi] job_id 16884275 proc_lsst.multi INFO: [multi] job_id 16884275 2024-03-07 03:37:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:37:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884275 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884275 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16884275 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 03:37:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 03:37:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:37:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:31 proc_lsst.multi:170 [INFO] [multi] job_id 16884276 proc_lsst.multi INFO: [multi] job_id 16884276 2024-03-07 03:37:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:37:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884276 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884276 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16884276 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 03:37:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 03:37:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:37:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884277 proc_lsst.multi INFO: [multi] job_id 16884277 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884277 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884277 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16884277 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884278 proc_lsst.multi INFO: [multi] job_id 16884278 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884278 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884278 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16884278 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884279 proc_lsst.multi INFO: [multi] job_id 16884279 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884279 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884279 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16884279 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884280 proc_lsst.multi INFO: [multi] job_id 16884280 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884280 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884280 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16884280 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884281 proc_lsst.multi INFO: [multi] job_id 16884281 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884281 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884281 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16884281 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884282 proc_lsst.multi INFO: [multi] job_id 16884282 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884282 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884282 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16884282 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884283 proc_lsst.multi INFO: [multi] job_id 16884283 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884283 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884283 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16884283 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884284 proc_lsst.multi INFO: [multi] job_id 16884284 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884284 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884284 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16884284 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:32 proc_lsst.multi:170 [INFO] [multi] job_id 16884285 proc_lsst.multi INFO: [multi] job_id 16884285 2024-03-07 03:37:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 03:37:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884285 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884285 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16884285 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 03:37:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 03:37:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:33 proc_lsst.multi:170 [INFO] [multi] job_id 16884286 proc_lsst.multi INFO: [multi] job_id 16884286 2024-03-07 03:37:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 03:37:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884286 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884286 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16884286 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 03:37:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:33 proc_lsst.multi:170 [INFO] [multi] job_id 16884287 proc_lsst.multi INFO: [multi] job_id 16884287 2024-03-07 03:37:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 03:37:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884287 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884287 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16884287 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 03:37:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:33 proc_lsst.multi:170 [INFO] [multi] job_id 16884288 proc_lsst.multi INFO: [multi] job_id 16884288 2024-03-07 03:37:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 03:37:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884288 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884288 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16884288 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 03:37:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:33 proc_lsst.multi:170 [INFO] [multi] job_id 16884289 proc_lsst.multi INFO: [multi] job_id 16884289 2024-03-07 03:37:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16884289 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16884289 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16884289 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-07 03:37:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:33 proc_lsst.multi:170 [INFO] [multi] job_id 16884290 proc_lsst.multi INFO: [multi] job_id 16884290 2024-03-07 03:37:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:33 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884290 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884290 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16884290 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-07 03:37:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:33 proc_lsst.multi:170 [INFO] [multi] job_id 16884291 proc_lsst.multi INFO: [multi] job_id 16884291 2024-03-07 03:37:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:33 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884291 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884291 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16884291 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 33, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/32 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 33 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 29 more blocks parsl.executors.status_handling INFO: Scaling out by 29 blocks parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-07 03:37:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:33 proc_lsst.multi:170 [INFO] [multi] job_id 16884292 proc_lsst.multi INFO: [multi] job_id 16884292 2024-03-07 03:37:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:33 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884292 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884292 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16884292 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-07 03:37:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:34 proc_lsst.multi:170 [INFO] [multi] job_id 16884293 proc_lsst.multi INFO: [multi] job_id 16884293 2024-03-07 03:37:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:34 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884293 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884293 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16884293 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-07 03:37:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:34 proc_lsst.multi:170 [INFO] [multi] job_id 16884294 proc_lsst.multi INFO: [multi] job_id 16884294 2024-03-07 03:37:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:34 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16884294 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16884294 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16884294 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-07 03:37:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:34 proc_lsst.multi:170 [INFO] [multi] job_id 16884295 proc_lsst.multi INFO: [multi] job_id 16884295 2024-03-07 03:37:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884295 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884295 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16884295 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-07 03:37:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:34 proc_lsst.multi:170 [INFO] [multi] job_id 16884296 proc_lsst.multi INFO: [multi] job_id 16884296 2024-03-07 03:37:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884296 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884296 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16884296 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-07 03:37:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:34 proc_lsst.multi:170 [INFO] [multi] job_id 16884297 proc_lsst.multi INFO: [multi] job_id 16884297 2024-03-07 03:37:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884297 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884297 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16884297 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-07 03:37:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:35 proc_lsst.multi:170 [INFO] [multi] job_id 16884298 proc_lsst.multi INFO: [multi] job_id 16884298 2024-03-07 03:37:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884298 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884298 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16884298 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-07 03:37:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:35 proc_lsst.multi:170 [INFO] [multi] job_id 16884299 proc_lsst.multi INFO: [multi] job_id 16884299 2024-03-07 03:37:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884299 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884299 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16884299 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-07 03:37:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:35 proc_lsst.multi:170 [INFO] [multi] job_id 16884300 proc_lsst.multi INFO: [multi] job_id 16884300 2024-03-07 03:37:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:37:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884300 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884300 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16884300 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-07 03:37:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:35 proc_lsst.multi:170 [INFO] [multi] job_id 16884301 proc_lsst.multi INFO: [multi] job_id 16884301 2024-03-07 03:37:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:37:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884301 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884301 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16884301 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-07 03:37:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:35 proc_lsst.multi:170 [INFO] [multi] job_id 16884302 proc_lsst.multi INFO: [multi] job_id 16884302 2024-03-07 03:37:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:37:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884302 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884302 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16884302 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-07 03:37:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 03:37:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:36 proc_lsst.multi:170 [INFO] [multi] job_id 16884303 proc_lsst.multi INFO: [multi] job_id 16884303 2024-03-07 03:37:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:37:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:36 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884303 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884303 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16884303 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-07 03:37:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:36 proc_lsst.multi:170 [INFO] [multi] job_id 16884304 proc_lsst.multi INFO: [multi] job_id 16884304 2024-03-07 03:37:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:37:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:36 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884304 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884304 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16884304 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-07 03:37:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:36 proc_lsst.multi:170 [INFO] [multi] job_id 16884305 proc_lsst.multi INFO: [multi] job_id 16884305 2024-03-07 03:37:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:37:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:36 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884305 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884305 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16884305 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-07 03:37:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 03:37:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:37 proc_lsst.multi:170 [INFO] [multi] job_id 16884306 proc_lsst.multi INFO: [multi] job_id 16884306 2024-03-07 03:37:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:37:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884306 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884306 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16884306 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-07 03:37:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:37 proc_lsst.multi:170 [INFO] [multi] job_id 16884307 proc_lsst.multi INFO: [multi] job_id 16884307 2024-03-07 03:37:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:37:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884307 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884307 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16884307 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-07 03:37:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:37 proc_lsst.multi:170 [INFO] [multi] job_id 16884308 proc_lsst.multi INFO: [multi] job_id 16884308 2024-03-07 03:37:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:37:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884308 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884308 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16884308 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-07 03:37:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:37 proc_lsst.multi:170 [INFO] [multi] job_id 16884309 proc_lsst.multi INFO: [multi] job_id 16884309 2024-03-07 03:37:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:37:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884309 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884309 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16884309 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-07 03:37:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 03:37:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:38 proc_lsst.multi:170 [INFO] [multi] job_id 16884310 proc_lsst.multi INFO: [multi] job_id 16884310 2024-03-07 03:37:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:37:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884310 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884310 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16884310 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-07 03:37:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:38 proc_lsst.multi:170 [INFO] [multi] job_id 16884311 proc_lsst.multi INFO: [multi] job_id 16884311 2024-03-07 03:37:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:37:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884311 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884311 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16884311 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-07 03:37:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:38 proc_lsst.multi:170 [INFO] [multi] job_id 16884312 proc_lsst.multi INFO: [multi] job_id 16884312 2024-03-07 03:37:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:37:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884312 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884312 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16884312 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-07 03:37:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:38 proc_lsst.multi:170 [INFO] [multi] job_id 16884313 proc_lsst.multi INFO: [multi] job_id 16884313 2024-03-07 03:37:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:37:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884313 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884313 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16884313 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-07 03:37:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 03:37:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:39 proc_lsst.multi:170 [INFO] [multi] job_id 16884314 proc_lsst.multi INFO: [multi] job_id 16884314 2024-03-07 03:37:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16884314 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16884314 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16884314 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-07 03:37:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:39 proc_lsst.multi:170 [INFO] [multi] job_id 16884315 proc_lsst.multi INFO: [multi] job_id 16884315 2024-03-07 03:37:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884315 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884315 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16884315 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-07 03:37:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:40 proc_lsst.multi:170 [INFO] [multi] job_id 16884316 proc_lsst.multi INFO: [multi] job_id 16884316 2024-03-07 03:37:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884316 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884316 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16884316 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-07 03:37:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:40 proc_lsst.multi:170 [INFO] [multi] job_id 16884317 proc_lsst.multi INFO: [multi] job_id 16884317 2024-03-07 03:37:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884317 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884317 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16884317 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-07 03:37:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:40 proc_lsst.multi:170 [INFO] [multi] job_id 16884318 proc_lsst.multi INFO: [multi] job_id 16884318 2024-03-07 03:37:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884318 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884318 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16884318 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-07 03:37:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:40 proc_lsst.multi:170 [INFO] [multi] job_id 16884319 proc_lsst.multi INFO: [multi] job_id 16884319 2024-03-07 03:37:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884319 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884319 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16884319 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-07 03:37:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 03:37:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 03:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 03:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54744 --result_port=54487 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 03:37:41 proc_lsst.multi:170 [INFO] [multi] job_id 16884320 proc_lsst.multi INFO: [multi] job_id 16884320 2024-03-07 03:37:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 03:37:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 03:37:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16884320 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16884320 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16884320 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 37 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a10b893c16f727d279b81013562d959 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash c000c493180d38cfca78582ff6fe4f13 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/deffdf84-00f5-42ac-8b35-fbbf1e104a66_isr_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/deffdf84-00f5-42ac-8b35-fbbf1e104a66_isr_1124087_18.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48b254d78dcb253b4cebf5f9e381ddc0 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash b8eff63a3e794af3ec256c8a9c2c6540 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/47702bda-15dc-4cd8-a515-57c309d9c787_characterizeImage_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/47702bda-15dc-4cd8-a515-57c309d9c787_characterizeImage_1123958_58.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/d2c2718b-8f73-4be9-8e85-8e62e63eb887_isr_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/d2c2718b-8f73-4be9-8e85-8e62e63eb887_isr_1123958_58.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 52 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 53 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05f2c881a3b57093b6f4c74e78c83d90 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash d88c07790d02664126770e52548bb35b parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cebac510-a288-4826-bc34-6a922b459833_characterizeImage_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cebac510-a288-4826-bc34-6a922b459833_characterizeImage_1123958_5.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4817615e-f71a-4a50-a411-5c007f00dea6_isr_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4817615e-f71a-4a50-a411-5c007f00dea6_isr_1123958_5.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 134f356d411a0c9582b172c36512945c with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 56b38d1d95ca4b0219ac1465852333b7 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/27399d5f-be61-4071-a646-a16317e6e2cd_characterizeImage_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/27399d5f-be61-4071-a646-a16317e6e2cd_characterizeImage_1123956_4.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/07fa1cc3-6bfd-4b7c-a3aa-e9d732e49d85_isr_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/07fa1cc3-6bfd-4b7c-a3aa-e9d732e49d85_isr_1123956_4.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 1/61 running/pending blocks, and 57 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d419e9031b8fd035a022a3f24428fd2 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 5c21cec00f841413205163a026078beb parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5433b8da-157d-42b0-b61b-ceb2330403d6_isr_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5433b8da-157d-42b0-b61b-ceb2330403d6_isr_1124087_2.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3cfcd9b4912dcd1f2b413270a1b16e3 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 5b19b8f064e4dfc9c90d63e2eaba1589 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2a17e544-37ae-45b6-b64a-6f1e6559d00f_characterizeImage_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2a17e544-37ae-45b6-b64a-6f1e6559d00f_characterizeImage_1123954_17.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b74acd28-6e1a-48aa-aed2-d94a85898bf2_isr_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b74acd28-6e1a-48aa-aed2-d94a85898bf2_isr_1123954_17.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76165894b28faeb1b59c302f5ef8d3f9 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash aac664482a6ed89b99414f36541e1eff parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/aa0c26fe-e098-46a4-8c3d-4555f11119b7_characterizeImage_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/aa0c26fe-e098-46a4-8c3d-4555f11119b7_characterizeImage_1123958_56.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f69a42c2-9d83-48b8-9683-de57ccc62b8b_isr_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f69a42c2-9d83-48b8-9683-de57ccc62b8b_isr_1123958_56.stderr 2024-03-07 03:38:01 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f29965e14a740c0801332fc923976d74 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 56dd8afe4b248f4a7df2ec6c66780e85 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/4889cdd0-26b7-4f93-a092-69a1685513f6_characterizeImage_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/4889cdd0-26b7-4f93-a092-69a1685513f6_characterizeImage_1123954_30.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/36e1780b-b0f5-4ce9-bc6d-fd99566bca1c_isr_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/36e1780b-b0f5-4ce9-bc6d-fd99566bca1c_isr_1123954_30.stderr 2024-03-07 03:38:01 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcdde0a06a72bf133f77cca8dd5599e5 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash c61a15b407efe1a18fa6bdaf28706531 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca567a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/42c12db6-cbfd-4d4b-8384-b1fbdd52e650_isr_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/42c12db6-cbfd-4d4b-8384-b1fbdd52e650_isr_1124087_21.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f546341c10ed6cdadf122505ffecb30f with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 5ec8a93091cc7028539489f31e6eae2a parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c1925e2b-76a4-4603-a9d2-472a6b3e8072_isr_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c1925e2b-76a4-4603-a9d2-472a6b3e8072_isr_1124087_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:01 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 863e1870c4eea825784e9a5522dd1206 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash a95c3e7506ea53d1302d0fa7cba34f79 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae50c8de-094a-42de-9567-f76b087f2ecc_characterizeImage_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae50c8de-094a-42de-9567-f76b087f2ecc_characterizeImage_1123956_30.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9c4bfa0-9c9e-402b-8999-4ec3053e671a_isr_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9c4bfa0-9c9e-402b-8999-4ec3053e671a_isr_1123956_30.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fb168998cb2597abdaba91dc1ea499b with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 565abf8e8232ac0badb4aed8580bf732 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/68922a5c-4198-4779-93cc-7572bdf7926d_characterizeImage_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/68922a5c-4198-4779-93cc-7572bdf7926d_characterizeImage_1123958_30.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7b02598-1ea9-4477-8580-0402e3390409_isr_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7b02598-1ea9-4477-8580-0402e3390409_isr_1123958_30.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59cd9364dda52a25883bf783c803bd00 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 5dc484c3c81d180d4ede135c752173ef parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/9aa81ef7-e9ef-4eb9-9813-036a4592c2fd_characterizeImage_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/9aa81ef7-e9ef-4eb9-9813-036a4592c2fd_characterizeImage_1123958_20.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c315e271-8e99-44fd-85d5-1731dec810ca_isr_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c315e271-8e99-44fd-85d5-1731dec810ca_isr_1123958_20.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ffdf90bd3712efedb1d830d5cba9b61 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash c95fc0757c2907b22b7a9364616c10b8 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/76a75126-88d9-4a0a-b861-d87131eddb56_characterizeImage_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/76a75126-88d9-4a0a-b861-d87131eddb56_characterizeImage_1123956_40.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2357d866-f938-4308-916f-07ef828d390a_isr_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2357d866-f938-4308-916f-07ef828d390a_isr_1123956_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:01 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6204906ff8bb2d3aadc50ed00ba7aa8 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash ace6588315cc672918fc1d7c250d1534 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6cb108ef-498b-4386-9d92-714ca362b2b9_characterizeImage_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6cb108ef-498b-4386-9d92-714ca362b2b9_characterizeImage_1123956_29.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/444f0127-f829-4c08-99fb-7e7291bdb8fe_isr_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/444f0127-f829-4c08-99fb-7e7291bdb8fe_isr_1123956_29.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaaf57570add6f9dadf7c010a1833384 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 3e4814f951c8d821f7c5797973a54329 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a404a1f8-5cdf-4b43-ac7f-8f06b3906581_characterizeImage_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a404a1f8-5cdf-4b43-ac7f-8f06b3906581_characterizeImage_1123954_3.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/059036a3-e5b7-4d62-ab89-76aaefd08e78_isr_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/059036a3-e5b7-4d62-ab89-76aaefd08e78_isr_1123954_3.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1394271a539e83f1a0ca3435c2d5d5c with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash d3cf38393958cc041f2f07fa13db966a parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/67efeb9f-a7f9-4e65-a2b6-528e96ed477d_characterizeImage_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/67efeb9f-a7f9-4e65-a2b6-528e96ed477d_characterizeImage_1123954_47.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1d96223b-bba7-44a5-897d-8bac30e06c30_isr_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1d96223b-bba7-44a5-897d-8bac30e06c30_isr_1123954_47.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2304ca9e7324eb88d90e6216c93c7187 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 8eab558966ecefd80044f943556fe898 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5a6662e0-7ba1-47b8-9674-011f3424737f_isr_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5a6662e0-7ba1-47b8-9674-011f3424737f_isr_1124087_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:02 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9996f2bc5348d9b61db7ee03b4062146 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash f273f7e2b31f124296b8ab2df5bda16f parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7ef3ea54-0f16-4711-8c9d-74d628272c1b_characterizeImage_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7ef3ea54-0f16-4711-8c9d-74d628272c1b_characterizeImage_1123956_46.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b9f16caf-8eca-46c8-b9e8-413ae3a66352_isr_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b9f16caf-8eca-46c8-b9e8-413ae3a66352_isr_1123956_46.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3388f3ab7306fc374b037a9800a7f560 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash ae5040ca36b20d7f0c7b23dfd71437e2 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6095d215-259e-462b-a873-bfea51af6078_characterizeImage_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6095d215-259e-462b-a873-bfea51af6078_characterizeImage_1123956_26.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/14d83e19-a4cd-4fc9-8146-06618addd296_isr_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/14d83e19-a4cd-4fc9-8146-06618addd296_isr_1123956_26.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0e4b688a1392b19c00cd7fa486534d0 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash e3b07580d0c5fb9745a5e2890346d7f4 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/bebccd5e-ecb1-46f1-9340-4c399de71118_characterizeImage_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/bebccd5e-ecb1-46f1-9340-4c399de71118_characterizeImage_1123958_49.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/76727476-967f-49df-93f6-27c5858e81ca_isr_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/76727476-967f-49df-93f6-27c5858e81ca_isr_1123958_49.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bfb5560364f3a072bbc3fee8c64c5a1 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 08b714b37b9c788db119d665e096d5d6 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c3554b5e-713f-47ae-92d8-10cb4c06fa3d_characterizeImage_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c3554b5e-713f-47ae-92d8-10cb4c06fa3d_characterizeImage_1123954_42.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/2f3b3363-0375-4008-a158-25a413c46091_isr_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/2f3b3363-0375-4008-a158-25a413c46091_isr_1123954_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:02 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6d21635e6b581ecd35261c4dd3eb0b5 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 03c5fa55de0adea66ef513e35166dc4c parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a759a67b-a71b-4b97-b25a-00062f320c6d_isr_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a759a67b-a71b-4b97-b25a-00062f320c6d_isr_1124087_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:03 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:03 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:03 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:03 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 198de757e242dccebf51aea79eca571c with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 0850f13e5429f37a23fb6811ed0bf6da parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c52a730f-8427-4434-a78a-c64ba0c6e4ab_characterizeImage_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c52a730f-8427-4434-a78a-c64ba0c6e4ab_characterizeImage_1123956_37.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e96c1ccd-522a-4bf0-8cc8-2cad27e4d914_isr_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e96c1ccd-522a-4bf0-8cc8-2cad27e4d914_isr_1123956_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:04 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 665cc665b7adfb50e0252e023e329db5 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash aeb364e008e6bf25ba4cd7d79cda550e parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0bbcb0dc-21f8-42e7-9ebb-d10fe8b03feb_characterizeImage_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0bbcb0dc-21f8-42e7-9ebb-d10fe8b03feb_characterizeImage_1123954_46.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/53dc026f-ee2a-4c19-9d53-8fbad91e50cf_isr_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/53dc026f-ee2a-4c19-9d53-8fbad91e50cf_isr_1123954_46.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be135a1104d723e74b08a454a84dde05 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 20719a6f8a9b30aad5c86ae11b0440ae parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2d477768-1e22-4c31-9a33-ad5f68d3f0c7_characterizeImage_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2d477768-1e22-4c31-9a33-ad5f68d3f0c7_characterizeImage_1123956_10.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/5b3e0b97-aeff-4551-ad9c-20b1a1bfbf35_isr_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/5b3e0b97-aeff-4551-ad9c-20b1a1bfbf35_isr_1123956_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:04 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee6cd5da7519562b3cef88964b96c64 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash b74fd8f75b5431659309b36ca78b18f2 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f120bacc-f06e-4222-ae7d-53219df6161d_characterizeImage_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f120bacc-f06e-4222-ae7d-53219df6161d_characterizeImage_1123958_44.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/12ed3dc4-12a6-4d09-8dec-84d117145150_isr_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/12ed3dc4-12a6-4d09-8dec-84d117145150_isr_1123958_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:04 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3077b05e85fa238f2d463489248ed033 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 13b7ece65f0f9218fd3707b9dcaeddb1 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/6e09f4ba-182e-440f-81f1-fbd8b6f84976_characterizeImage_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/6e09f4ba-182e-440f-81f1-fbd8b6f84976_characterizeImage_1123958_3.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/fc10b91b-c5f2-4ac1-b309-db774121335b_isr_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/fc10b91b-c5f2-4ac1-b309-db774121335b_isr_1123958_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:04 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:05 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:05 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:05 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e904c47621395765946bb1fb5a94f8dc with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 0552d2669e6df8dc2df686791c4377e4 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cf15b91b-1e78-46c6-ad7d-c025ceeea493_characterizeImage_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cf15b91b-1e78-46c6-ad7d-c025ceeea493_characterizeImage_1123958_10.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/96edb08d-5ed7-40c3-9918-5edd9450eb9e_isr_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/96edb08d-5ed7-40c3-9918-5edd9450eb9e_isr_1123958_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:05 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:06 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:06 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:06 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:07 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:07 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:07 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:08 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:08 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:08 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d20ce38786171aabdda740a8b4b0e5fe with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 40b929bd068debc3260d16e1f773b51e parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7ed490b3-24a1-463d-9c6b-ed9853c536d2_characterizeImage_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7ed490b3-24a1-463d-9c6b-ed9853c536d2_characterizeImage_1123954_36.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b73f51a9-6d4c-4bf0-8eec-9074b0e3120e_isr_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b73f51a9-6d4c-4bf0-8eec-9074b0e3120e_isr_1123954_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:09 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:09 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:09 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:09 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:09 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:10 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:10 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c827e350f07b2edf98dcd89810df9b5c with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 5f0c5827479050d8cba88b0d7c8f742b parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e8044f82-a42d-49bb-a979-b27622003b8d_characterizeImage_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e8044f82-a42d-49bb-a979-b27622003b8d_characterizeImage_1123958_37.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/04f303db-d9a2-4dfd-82e7-51bf995199ce_isr_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/04f303db-d9a2-4dfd-82e7-51bf995199ce_isr_1123958_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:10 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:10 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:11 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:11 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:11 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:11 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:11 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:12 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:12 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:12 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:13 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c2761e3ac35230e8976709ec98197a5 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash e104851f4ad85a20b1ba49f8a1407deb parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/13ac5d55-50db-4aba-bdd9-ad1b0fc96349_characterizeImage_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/13ac5d55-50db-4aba-bdd9-ad1b0fc96349_characterizeImage_1123956_25.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/bb7e261e-09cb-4998-91ad-01c799f3d543_isr_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/bb7e261e-09cb-4998-91ad-01c799f3d543_isr_1123956_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:13 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac2e0e1ff991e3bd5d5e52e5fd1b8153 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 8609cdf91392899dc6017dff86ec9c4d parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2c6e1906-4e3b-432c-adb0-27b949704996_characterizeImage_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2c6e1906-4e3b-432c-adb0-27b949704996_characterizeImage_1123954_13.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e991e8cc-7f76-467b-9603-33d370b7d626_isr_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e991e8cc-7f76-467b-9603-33d370b7d626_isr_1123954_13.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cffcde1ddfcec7a5c3c5593dc773d46a with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 5f3c9521f8a73aaab6b85a256d2c4da7 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/72744fcc-a947-4845-950e-51d0961c844d_characterizeImage_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/72744fcc-a947-4845-950e-51d0961c844d_characterizeImage_1123958_23.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/061e7bfd-b5fe-4e62-9d22-a6690810e704_isr_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/061e7bfd-b5fe-4e62-9d22-a6690810e704_isr_1123958_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:13 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce268f63ef3930dce650a2378295ba94 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 96de4f8b253dd9a2bed047399aad20bb parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:14 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b271d02d-e20f-48c7-9a7d-9de8ff0de76f_characterizeImage_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b271d02d-e20f-48c7-9a7d-9de8ff0de76f_characterizeImage_1123956_17.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/11b28325-5d46-4b09-a85d-d849b17899ef_isr_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/11b28325-5d46-4b09-a85d-d849b17899ef_isr_1123956_17.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a43d3fe343d4dfd54edaf0e6890e79cc with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash f852b6b7743d2acb47ad78e34abec803 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/81ea7ca2-6583-4d38-865a-01ba50492949_characterizeImage_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/81ea7ca2-6583-4d38-865a-01ba50492949_characterizeImage_1123956_45.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9231208-9864-4945-8787-2b52d4a587fd_isr_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9231208-9864-4945-8787-2b52d4a587fd_isr_1123956_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:14 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c14c43d53b01a0d45c3c4b1dd3ee42d3 with result from task 216 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:14 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash cf93be5b5f6be3415f03e3ce320e0f96 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a416787d-5e1d-467f-a6b5-3c83331f763f_isr_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a416787d-5e1d-467f-a6b5-3c83331f763f_isr_1124087_4.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a91342ab620245a7213d61e8266b0b9 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 4cda3dff1034ed6eb3cdd5f0bf7e8a47 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f72e375b-4525-4847-bb05-f487da911ffc_isr_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f72e375b-4525-4847-bb05-f487da911ffc_isr_1124087_39.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e8cf7843bb0aa525a0e8ad1e7da1237 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash a52f7f454350b07705c818f2aadbd17d parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/25dbaed4-cd2f-4827-bfa4-8192b07b4931_isr_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/25dbaed4-cd2f-4827-bfa4-8192b07b4931_isr_1124087_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:15 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 385ccff954400de553af0954c56a1735 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 64dd7256f42c78630eacc814d4d1082a parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/b8ce25b6-6f9c-48e9-ae23-c547289ffc3d_characterizeImage_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/b8ce25b6-6f9c-48e9-ae23-c547289ffc3d_characterizeImage_1123954_19.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d19759be-46b4-45f9-bcfb-4ddc0e306a50_isr_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d19759be-46b4-45f9-bcfb-4ddc0e306a50_isr_1123954_19.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fad76c6f294cd534fd819c3579d952ab with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 3aca6d2112cc94aef2bdde2eadbfffdc parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f289007c-b4e4-4272-bc29-6a3df2ab3cd5_characterizeImage_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f289007c-b4e4-4272-bc29-6a3df2ab3cd5_characterizeImage_1123958_28.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1b43bcb3-08f6-4bd6-9ec5-cb13cf7e6b9a_isr_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1b43bcb3-08f6-4bd6-9ec5-cb13cf7e6b9a_isr_1123958_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:15 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b463e0a2821a04cfa3fa97a24a1e72a with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 23815e0fb2384d49933fbbc93723d527 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/5a84869e-a9fb-4cf1-9b36-8abbf1781aaf_characterizeImage_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/5a84869e-a9fb-4cf1-9b36-8abbf1781aaf_characterizeImage_1123958_22.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f591a904-2988-4dbc-8811-de243337a42b_isr_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f591a904-2988-4dbc-8811-de243337a42b_isr_1123958_22.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 502b2706cd419e0ececb23981a326a76 with result from task 219 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:16 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash bd845f52772843c76417c5a384d2ccc0 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ede9e5c6-2d9f-491c-b84a-76b2dbfac949_isr_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ede9e5c6-2d9f-491c-b84a-76b2dbfac949_isr_1124087_8.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90b0986a713ffd9a212a134aef97ada6 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash ea0df0b9aa43264addbc5d651ed98961 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/19872edb-fb88-4fec-bb9c-779d8c7ae462_characterizeImage_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/19872edb-fb88-4fec-bb9c-779d8c7ae462_characterizeImage_1123954_50.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9ee6315b-35fb-41d4-898b-7aa8a38cbb91_isr_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9ee6315b-35fb-41d4-898b-7aa8a38cbb91_isr_1123954_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:16 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a19c87f92a7add8d3a072124399e602 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash ff146db96e85f01d98079834218ea0a3 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83184d14-5f6a-452c-b57a-d046a6f331aa_characterizeImage_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83184d14-5f6a-452c-b57a-d046a6f331aa_characterizeImage_1123956_7.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/324054db-c529-4156-b612-b30585a9b788_isr_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/324054db-c529-4156-b612-b30585a9b788_isr_1123956_7.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8164237625964d1f0d37b98e2318167f with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 202f87230bbf2c309e5edc9856b456e2 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1328b0e2-485e-4f73-89f9-734cbbdf36b1_isr_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1328b0e2-485e-4f73-89f9-734cbbdf36b1_isr_1124087_6.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff7c7aa561d651368dd6b1380d7a4a8e with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash a515cdccae2b9ce0ea2a92a449e4b3cc parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a811ac9-f3aa-4151-9ac7-b011984238f6_isr_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a811ac9-f3aa-4151-9ac7-b011984238f6_isr_1124087_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:16 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 356d61e3ba120ff0ae876056a3dab10c with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 7805762b40c9a85ff130c9ff1d0580d6 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7bfac662-56dc-4c61-9402-2a4359ad8299_characterizeImage_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7bfac662-56dc-4c61-9402-2a4359ad8299_characterizeImage_1123954_16.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/705596b8-29f6-4d5c-870f-0d430e04e79e_isr_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/705596b8-29f6-4d5c-870f-0d430e04e79e_isr_1123954_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:16 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b1bac65126e3219caee2305cedb85b0 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash ca278d9909edd00f367e9d212cdb0cdb parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/001d1e48-7220-411d-9761-3d6b6cc744b9_characterizeImage_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/001d1e48-7220-411d-9761-3d6b6cc744b9_characterizeImage_1123956_35.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8bc78721-19c7-41ae-a987-1d7f0c3f2513_isr_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8bc78721-19c7-41ae-a987-1d7f0c3f2513_isr_1123956_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:17 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75f8136c5049ef586daa72a99d5be832 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 1009efc93f27e946726524387e062c17 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96bdec32-7a93-4582-8071-9b3fb86e96ff_characterizeImage_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96bdec32-7a93-4582-8071-9b3fb86e96ff_characterizeImage_1123956_11.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/228cb8fd-0757-49e8-81a5-b0306688f448_isr_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/228cb8fd-0757-49e8-81a5-b0306688f448_isr_1123956_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:17 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:17 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07c32a00891bd81b7211772f820f8cf4 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 45022d91f992ecbb1a70939e8d42610f parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a65da1ef-c0d6-470a-8a08-284a99d406dc_characterizeImage_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a65da1ef-c0d6-470a-8a08-284a99d406dc_characterizeImage_1123958_13.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/0f143f63-a874-4c29-9e56-3ee4c053ef42_isr_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/0f143f63-a874-4c29-9e56-3ee4c053ef42_isr_1123958_13.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1670d30648702843e2f9e90dc67dc7a2 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 9953cc9969843c73e6ef533712896812 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b72a3a03-1be5-4f4b-83e0-8ea9c57334da_characterizeImage_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b72a3a03-1be5-4f4b-83e0-8ea9c57334da_characterizeImage_1123958_1.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f41214ad-b66b-426d-b5e4-1c7a6ca49c89_isr_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f41214ad-b66b-426d-b5e4-1c7a6ca49c89_isr_1123958_1.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c420fbe6e8ad1514f316754a8f3b01 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 07e63588df970413f34c38152add60dd parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dfd1ea59-1ae0-4b08-9044-221861ee6fd7_isr_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dfd1ea59-1ae0-4b08-9044-221861ee6fd7_isr_1123956_2.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf6a1da334d06a67601935634720893 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 1f7f315e411f79e6dd2ad7cbbf332a0e parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/1cca87b8-5e76-4ed4-95a4-06ef65fab94f_characterizeImage_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/1cca87b8-5e76-4ed4-95a4-06ef65fab94f_characterizeImage_1123958_36.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3e464a58-173e-4c3d-acd4-08d8b3fbb6a1_isr_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3e464a58-173e-4c3d-acd4-08d8b3fbb6a1_isr_1123958_36.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e08887d3dd37a6cf7546eca6c91b03f4 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash a393f85537464a1abed2e760375528a5 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0e8dbd00-7db5-497e-a65a-46e967057f96_characterizeImage_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0e8dbd00-7db5-497e-a65a-46e967057f96_characterizeImage_1123956_3.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/56de81a0-049f-4c03-af45-f0dd1a38afdb_isr_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/56de81a0-049f-4c03-af45-f0dd1a38afdb_isr_1123956_3.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 258b2937a8dca9c14e7d14b8eef57c27 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash b4cbb7ee26d3cfb1cd28fd1145a9f6c7 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8900e632-f597-4d29-96f8-5ebbb81dbd96_characterizeImage_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8900e632-f597-4d29-96f8-5ebbb81dbd96_characterizeImage_1123958_57.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/02f96563-44f0-415b-874c-374712afd587_isr_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/02f96563-44f0-415b-874c-374712afd587_isr_1123958_57.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e5fb40538fbc1430356832a09d982b1 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 79660152977a0c14df444f7ebb7cad75 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1a90975e-bcf4-44a0-a9f8-8ee889807b33_isr_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1a90975e-bcf4-44a0-a9f8-8ee889807b33_isr_1124087_54.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 617d824d3406c0473ccf97d796b74f24 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 915797eb43c9c476109dcb9fe6070674 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/35d36398-95c8-4b53-8e5e-4923013595ea_isr_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/35d36398-95c8-4b53-8e5e-4923013595ea_isr_1124087_5.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22588693ddc89b71be2a039081e4d6a6 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 553470b5f4d9ae60e6df7935880cea98 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca554e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ad567e8d-30c1-4b03-ba54-f527108cc039_isr_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/ad567e8d-30c1-4b03-ba54-f527108cc039_isr_1124087_40.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92ae52beddfbcebb9712b88db27b2cbf with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash ad6166e25c076303099816d1d40bb95c parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca551c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3196024-1417-4243-a66b-d1c651212e28_isr_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3196024-1417-4243-a66b-d1c651212e28_isr_1124087_23.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 810026c723624b0ab134440ab0d1ce61 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash d0157cc767ad343edd201bbd7978c5e3 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7f5965a2-75f2-4d4b-9ad5-a9fb67b0768f_characterizeImage_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7f5965a2-75f2-4d4b-9ad5-a9fb67b0768f_characterizeImage_1123958_33.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9b6afde-4899-432e-9d84-82bda0386f7d_isr_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9b6afde-4899-432e-9d84-82bda0386f7d_isr_1123958_33.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 275b76eb4e1ad94cdd2c61f56d0d8256 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 41b476119f954a316f3d659c153fdd3a parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7117b39f-998d-48df-89d6-78e7c5a8dcfb_characterizeImage_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7117b39f-998d-48df-89d6-78e7c5a8dcfb_characterizeImage_1123956_27.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9ceab93-00d6-4de2-977e-2535031b1926_isr_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e9ceab93-00d6-4de2-977e-2535031b1926_isr_1123956_27.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c5351e27fabc4678bd89fa5c922d32b with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash fec6ed580c9b9f9517e191b0899804dd parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f3e61f51-4378-462c-8329-b8250d5c5d77_characterizeImage_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f3e61f51-4378-462c-8329-b8250d5c5d77_characterizeImage_1123958_17.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/626d844d-b14b-444a-97d4-e8d9993815e0_isr_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/626d844d-b14b-444a-97d4-e8d9993815e0_isr_1123958_17.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3813805de8f8f1d57ae40fb1cda0ab13 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 749bfd5514d2addf53bfc74795b330bb parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ffe34f4d-6004-48e0-9e22-035d095815c7_characterizeImage_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ffe34f4d-6004-48e0-9e22-035d095815c7_characterizeImage_1123958_12.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f16fd25c-b29b-4619-b1ad-8d1da2120f7a_isr_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/f16fd25c-b29b-4619-b1ad-8d1da2120f7a_isr_1123958_12.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4476d8cdf3102144b1cbb64dbcc17d15 with result from task 33 parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:17 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash e89bad3fbb7a2001293003b481e85847 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bda18c61-7e4c-414d-9027-bbd0420675c9_isr_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bda18c61-7e4c-414d-9027-bbd0420675c9_isr_1124087_37.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 544df9b8b61367ee4b24782e0215fb46 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 6cba8c46a7457fd0a042afaff70a4077 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/542f0806-fa57-4df8-9745-3398559eadf6_characterizeImage_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/542f0806-fa57-4df8-9745-3398559eadf6_characterizeImage_1123958_15.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b327b194-d124-4e78-a21d-6e337ad73d26_isr_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b327b194-d124-4e78-a21d-6e337ad73d26_isr_1123958_15.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8ece6881f492064913f2b9f7bac43b5 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 85e9ad05348ce00d30b61fbf3e6c1612 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3a44df86-a44c-4e5c-b49f-1d55ec37a8da_characterizeImage_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3a44df86-a44c-4e5c-b49f-1d55ec37a8da_characterizeImage_1123958_7.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c0063737-b2f6-4bd2-b81e-353b158a0a3e_isr_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c0063737-b2f6-4bd2-b81e-353b158a0a3e_isr_1123958_7.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a6f2bffe028a51ea152729bb0daeb98 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash b392e08abf3b562aafe9e7defecbb6e1 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca544a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/90911a30-cf15-4550-ae21-05770920adcc_isr_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/90911a30-cf15-4550-ae21-05770920adcc_isr_1124087_43.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6848497815d68859de1bb6606b166c4 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash f2a5062f15bd579f4425f01dfce928b2 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/037568f8-0932-4467-bd82-93e364608cba_characterizeImage_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/037568f8-0932-4467-bd82-93e364608cba_characterizeImage_1123954_61.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a155b02c-df2d-4cf5-8398-9f710641f0a7_isr_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a155b02c-df2d-4cf5-8398-9f710641f0a7_isr_1123954_61.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28f5e3cb7a3c36a5447f3f79f66c8586 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash b4aa527b06aa66d0b41e4cf77de76d9f parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1acaae9e-4843-4a2c-bebe-1c77a8dd3a87_characterizeImage_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1acaae9e-4843-4a2c-bebe-1c77a8dd3a87_characterizeImage_1123954_48.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/829b6cb1-eee3-4530-bd61-0bd0e2795f76_isr_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/829b6cb1-eee3-4530-bd61-0bd0e2795f76_isr_1123954_48.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d921c308174a5050f59785c7bb61231 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash f894d4b850d9e994e0980e14cc6cb97d parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/eed52d84-5882-457e-8402-ab8c5a497f04_characterizeImage_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/eed52d84-5882-457e-8402-ab8c5a497f04_characterizeImage_1123958_11.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/ef4efb41-09ad-4c42-906c-1d1c6b84e5d0_isr_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/ef4efb41-09ad-4c42-906c-1d1c6b84e5d0_isr_1123958_11.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bd6afa36e946c3c32d6db8663077af0 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 66ad6fbc5f428cb1cb5eb9ab165fe04a parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f83735cd-a35d-4b6c-8596-fd11f2022e62_characterizeImage_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f83735cd-a35d-4b6c-8596-fd11f2022e62_characterizeImage_1123958_45.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/86db1407-c4a2-49a2-9d4b-90ddba3e1343_isr_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/86db1407-c4a2-49a2-9d4b-90ddba3e1343_isr_1123958_45.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28b9bd8f98819dd33a4de39eca78cb5 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash dafbac4285bbf0db6addc9dc9e7aab46 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b90c526d-b36c-419f-a549-f9a178673659_characterizeImage_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b90c526d-b36c-419f-a549-f9a178673659_characterizeImage_1123956_32.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65435dce-3ca4-48d1-90a5-8fe0851e1192_isr_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65435dce-3ca4-48d1-90a5-8fe0851e1192_isr_1123956_32.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad528cced7f02cc934ec4505a0d5026d with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 34627830226198aec8c42ab23a2d89dd parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/fa33193c-780a-40c7-8515-4cfec960a29e_isr_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/fa33193c-780a-40c7-8515-4cfec960a29e_isr_1124087_11.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fcf0adb39bc05a967e8ee6366f47811 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 81f87314cd2d2eea05ca4fe6ba13d991 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/961418db-2d8c-4744-930f-036a28ffde04_isr_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/961418db-2d8c-4744-930f-036a28ffde04_isr_1124087_3.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 901ed7baaaa98ec328d167e600a4af97 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 7149744b5401b989c635a544fcc93d85 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/23110368-8bbf-40dc-b491-01bbe4c8818f_isr_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/23110368-8bbf-40dc-b491-01bbe4c8818f_isr_1124087_20.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d043faaf116c552e3b87fe5fc9b2559c with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash d556701645b0eb4bcbfa87cdf609571c parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/75d0e32b-58be-4b60-8af4-b5f46714ff7f_isr_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/75d0e32b-58be-4b60-8af4-b5f46714ff7f_isr_1124087_36.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b81f1727353ac138923f4b8cef58c38f with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 154021f8851b3fa04995b32ae727503d parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9db5c473-f69f-47d2-b332-4b46ba0b6104_characterizeImage_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9db5c473-f69f-47d2-b332-4b46ba0b6104_characterizeImage_1123954_52.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/696d6c20-a2db-4ecb-8eea-7f66f390cc37_isr_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/696d6c20-a2db-4ecb-8eea-7f66f390cc37_isr_1123954_52.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62208f1ede0b3a22267e8a6c24051089 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash e2c5e36e40180559ebd0cc3ee3130303 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6b381df5-69a4-47b7-b459-4b6d1e4b84a3_characterizeImage_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6b381df5-69a4-47b7-b459-4b6d1e4b84a3_characterizeImage_1123954_7.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d84f8387-8865-4ca9-bc2c-88b1f6be760c_isr_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d84f8387-8865-4ca9-bc2c-88b1f6be760c_isr_1123954_7.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e496c40d2e34532e35b71ba66a8d45b0 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash a093b1c60091bf0c90d0596aef07fc35 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:18 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fd29101-c2f0-4080-99e3-969b0dce24fd_characterizeImage_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fd29101-c2f0-4080-99e3-969b0dce24fd_characterizeImage_1123954_1.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9c83036-a1d3-4523-bb31-4bc49fa1ba23_isr_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9c83036-a1d3-4523-bb31-4bc49fa1ba23_isr_1123954_1.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3e9d792fd2576479e6e12975159810a with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 9b2f2682a2d585d3d8ec67f6475f8d3c parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/877ecc48-0cf1-42e6-b692-177d52d4fb34_characterizeImage_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/877ecc48-0cf1-42e6-b692-177d52d4fb34_characterizeImage_1123956_52.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/4ec7e3d9-bceb-4ada-9459-1545f5fbaca3_isr_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/4ec7e3d9-bceb-4ada-9459-1545f5fbaca3_isr_1123956_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:18 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1159e99d8163a181c7ee11be882ee83d with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash f3bfc1dc081e278c9d433b830dc085a7 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/97739230-d1a1-4f2c-96f7-d3b430a19053_characterizeImage_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/97739230-d1a1-4f2c-96f7-d3b430a19053_characterizeImage_1123958_27.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4aac3d9-b30f-453f-b74d-c5cfa7b51164_isr_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4aac3d9-b30f-453f-b74d-c5cfa7b51164_isr_1123958_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f7ba0e5a66fbbb4781dcd6bfd46c734 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 144e86126e7076711615e6728a43e4b9 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3d2113f8-c13c-4ada-b7eb-aae81c13418b_characterizeImage_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3d2113f8-c13c-4ada-b7eb-aae81c13418b_characterizeImage_1123954_23.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfda0d70-d81e-4310-b263-693c0b1dfb57_isr_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfda0d70-d81e-4310-b263-693c0b1dfb57_isr_1123954_23.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4f5e4f6968e7ec9df02ff8c49163015 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 47b51ed80d86ec0e85a1f8a16cedcbc3 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b44a3808-6e1c-4449-8b28-367404cefc75_characterizeImage_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b44a3808-6e1c-4449-8b28-367404cefc75_characterizeImage_1123958_9.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9490ad4-55a3-4b20-a96a-cc0efcdf730c_isr_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a9490ad4-55a3-4b20-a96a-cc0efcdf730c_isr_1123958_9.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50a55e181fa574ef523efa6eba4d82c9 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 5077b83a1c2ad93fb9e16caedfcd81e3 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8b69b961-27e7-4f30-9b0c-9ae9f5e4627f_isr_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8b69b961-27e7-4f30-9b0c-9ae9f5e4627f_isr_1124087_29.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6981d26f36ac595f15bda165d279131b with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 0522f7977fc5c4597cd2e6a761d55873 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1289a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c7f40355-8c24-4eb6-b3f7-846fef8c1845_characterizeImage_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c7f40355-8c24-4eb6-b3f7-846fef8c1845_characterizeImage_1123956_41.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1da25323-a804-4f44-b102-016d96bfc1fd_isr_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1da25323-a804-4f44-b102-016d96bfc1fd_isr_1123956_41.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 789618aa73641cf9ac78b90a57b74d33 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash a3dc5697ee8f4a4bfbbdfb2ce1060ea3 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae3c5d37-483c-47c4-a495-79f0afae9d04_characterizeImage_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae3c5d37-483c-47c4-a495-79f0afae9d04_characterizeImage_1123956_8.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1c092786-b9d7-4cb5-88e3-b0dae5b27ebe_isr_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/1c092786-b9d7-4cb5-88e3-b0dae5b27ebe_isr_1123956_8.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f5677d106d5a91e253fd84452d2335b with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash db114faafb5b568b7d87f36f26b1c706 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1284a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a1d92bdd-13a8-4e86-b557-9f609e3c27c8_isr_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a1d92bdd-13a8-4e86-b557-9f609e3c27c8_isr_1124087_28.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cba48ade0d90681e53b970b3561c3ba2 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 8a1ff1ad54c9a5509ed7d083a77994f5 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fc301d8c-b866-4c66-8c8d-08daab99fcde_characterizeImage_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fc301d8c-b866-4c66-8c8d-08daab99fcde_characterizeImage_1123954_43.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e5182262-280b-46c4-9b60-aa1ef6cba5d9_isr_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e5182262-280b-46c4-9b60-aa1ef6cba5d9_isr_1123954_43.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e3f7bda762fb4076adf4bc0f0bb5001 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 206980ffa999d42b0a2af0956055c46c parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/65c9d02e-9918-49ca-8d86-1fce8a1ae631_characterizeImage_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/65c9d02e-9918-49ca-8d86-1fce8a1ae631_characterizeImage_1123958_31.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/517bb18e-e3e2-4f2e-9c3c-0f127a67b1c9_isr_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/517bb18e-e3e2-4f2e-9c3c-0f127a67b1c9_isr_1123958_31.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4af972ee46c663e55eb4b00132fa6a8e with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 522b25a7f14981f2d6a14d6f8c08009a parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a2b6e6c7-b247-4014-a376-ab4febdff69a_characterizeImage_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a2b6e6c7-b247-4014-a376-ab4febdff69a_characterizeImage_1123956_15.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/957150f1-f4ea-4c12-959a-2920ce07da99_isr_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/957150f1-f4ea-4c12-959a-2920ce07da99_isr_1123956_15.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b65860c8e18dc0f39a3fd31d4c592612 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 3f3472f926f1ebedf9f9f1268bd3a855 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/116c7e82-b5c3-41e1-90f2-d907191b13b7_characterizeImage_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/116c7e82-b5c3-41e1-90f2-d907191b13b7_characterizeImage_1123958_51.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c12ea14b-803b-4955-905b-25d671200046_isr_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c12ea14b-803b-4955-905b-25d671200046_isr_1123958_51.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83b810c3c05a47eefa59e2b55a42eef4 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 6495d905ba6d150395f6dd6be9a0ca54 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2283f775-7f0b-41da-9bd0-0a7bb0e14ed9_characterizeImage_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2283f775-7f0b-41da-9bd0-0a7bb0e14ed9_characterizeImage_1123956_38.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b67b6e0d-6c52-485e-8206-d781b12478ee_isr_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b67b6e0d-6c52-485e-8206-d781b12478ee_isr_1123956_38.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af1c5229cbdc342cc8285c3346401db9 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 23853043d7d639e9db29e25252c5510f parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/94a26a03-7fa5-434e-911b-00b7d7b892b4_characterizeImage_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/94a26a03-7fa5-434e-911b-00b7d7b892b4_characterizeImage_1123956_19.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a3d4f5b1-b01c-4a74-89cd-f77a5978f078_isr_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a3d4f5b1-b01c-4a74-89cd-f77a5978f078_isr_1123956_19.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf80dcc85132e64406825b17c0e2cca0 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 3874c8b99cd5ce0654066bdbd7db990a parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c1b5e331-343c-40ee-a234-fd7caa10750c_characterizeImage_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c1b5e331-343c-40ee-a234-fd7caa10750c_characterizeImage_1123958_54.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/db2535a6-119d-4326-82c3-248504abd2bf_isr_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/db2535a6-119d-4326-82c3-248504abd2bf_isr_1123958_54.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbbc5ce51646d4a2884dcd68c7509450 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash e30bf3af893191d555b0063c97e2f59d parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fb4c9005-ff31-4e3f-86a6-63827d39ffa5_characterizeImage_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fb4c9005-ff31-4e3f-86a6-63827d39ffa5_characterizeImage_1123954_21.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/987a884c-3dfa-48c4-a94b-7855b1df7611_isr_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/987a884c-3dfa-48c4-a94b-7855b1df7611_isr_1123954_21.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa0ab7d9a92783bac2efcddd35c7d5d7 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash f4813cf1b43fae9bd49dc6d620e73480 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8b9c434e-94d0-4a5b-ac4b-d34485f1fa2c_characterizeImage_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8b9c434e-94d0-4a5b-ac4b-d34485f1fa2c_characterizeImage_1123958_24.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9f27e7cc-f627-468f-b358-90af8c6b17ed_isr_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9f27e7cc-f627-468f-b358-90af8c6b17ed_isr_1123958_24.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d3a677eee9225f7b8263c1e31b93a99 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 8ca2cb24058a080aa282df8f5fdb6571 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/03d44e55-3e08-4779-897b-51b9dfbf42b3_characterizeImage_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/03d44e55-3e08-4779-897b-51b9dfbf42b3_characterizeImage_1123956_57.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65830161-a482-4857-b1f5-bfbbe154d842_isr_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/65830161-a482-4857-b1f5-bfbbe154d842_isr_1123956_57.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68f81449717d3ec91b189f2a36d7c25d with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 4f998674c2f1dcb933e571040ad11615 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d98bc6c4-eae6-40c2-ad6f-0c69dc0d48c9_characterizeImage_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d98bc6c4-eae6-40c2-ad6f-0c69dc0d48c9_characterizeImage_1123958_39.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c0fa8a0-2e25-4f86-934a-401be3d32ef7_isr_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c0fa8a0-2e25-4f86-934a-401be3d32ef7_isr_1123958_39.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 630b0ceecc2eee142bafe9366a8344da with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash d38f260afe21ef5f8f1360b491222803 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c50c0a67-3244-4790-a9f5-0f474ce03a6e_characterizeImage_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c50c0a67-3244-4790-a9f5-0f474ce03a6e_characterizeImage_1123954_40.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f91a55c-d08e-45b6-802e-2925ae0e0593_isr_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f91a55c-d08e-45b6-802e-2925ae0e0593_isr_1123954_40.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16e53b055975f3a48fa3674e5592a35c with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash a29c7f94e66b1b67e11a7eca27e760a2 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/81284995-f663-4c09-9a7b-c5fa00b28416_characterizeImage_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/81284995-f663-4c09-9a7b-c5fa00b28416_characterizeImage_1123954_29.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b795e720-acbc-440a-a558-b9573ebf638a_isr_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b795e720-acbc-440a-a558-b9573ebf638a_isr_1123954_29.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af7ba658275e9792722a2c31aa53a20e with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 3ff3465232b98d1e68709757d353b81a parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3d6c6545-174b-4113-934a-876f4dea32ca_isr_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3d6c6545-174b-4113-934a-876f4dea32ca_isr_1124087_32.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 685560191af8fc987295ca63046e6c04 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 7c33f0cd2e427bd189532b143f1a5b2c parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/e135b397-9812-45b5-bebb-e73e6c1bddaf_characterizeImage_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/e135b397-9812-45b5-bebb-e73e6c1bddaf_characterizeImage_1123956_21.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/6c9f88fe-abd1-4bb8-be36-db595f4e74e8_isr_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/6c9f88fe-abd1-4bb8-be36-db595f4e74e8_isr_1123956_21.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6539ae7d5bfc40e5255062eb4f2829ee with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash bae99d87cdf984b44621a63899e76f3e parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/9cd616a1-699c-4210-8046-037992974b74_isr_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/9cd616a1-699c-4210-8046-037992974b74_isr_1124087_55.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efd4517c5b1b3d2fec272a167e421f6c with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash a0b2884dfcf727b43e565a26211c3325 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6138ca77-60af-45e5-8503-b300e6cd0028_characterizeImage_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6138ca77-60af-45e5-8503-b300e6cd0028_characterizeImage_1123954_8.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/34b29c6e-028d-4d89-aee6-73f3a39dde98_isr_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/34b29c6e-028d-4d89-aee6-73f3a39dde98_isr_1123954_8.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e52044a6242894bba1152713dab478ee with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 358b9ee01929f6a95a8ea19df9a5f947 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/2287ae72-2f8f-48d5-b99c-a437ab82b3fd_isr_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/2287ae72-2f8f-48d5-b99c-a437ab82b3fd_isr_1124087_51.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51cd321a906597399b87b63c2c11317a with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 24b1ee619b42044e345b3849df8a3965 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8bf5fa77-1681-4e03-b2cd-247c411fd834_characterizeImage_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8bf5fa77-1681-4e03-b2cd-247c411fd834_characterizeImage_1123956_14.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c92b88a4-1075-494a-a1c3-1c884d7478aa_isr_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c92b88a4-1075-494a-a1c3-1c884d7478aa_isr_1123956_14.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c10e43bda3a9e943b199038fa8dbe5cd with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 0a5089ac34d921827cdeb741f7045c5c parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/da703fee-55ad-4028-ad47-fc2b316033ee_characterizeImage_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/da703fee-55ad-4028-ad47-fc2b316033ee_characterizeImage_1123958_8.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3bc21b3b-6267-43d5-b0e1-3761e38465b3_isr_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3bc21b3b-6267-43d5-b0e1-3761e38465b3_isr_1123958_8.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f520fc755979a4a2da3d22344ee67e with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 666235b7f1bf254e6fd9f376237c9c6b parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9f472724-4b53-49b1-ae6f-f947e5687769_characterizeImage_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9f472724-4b53-49b1-ae6f-f947e5687769_characterizeImage_1123954_33.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8879195d-4af4-4d19-a171-b27ac2a681c7_isr_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8879195d-4af4-4d19-a171-b27ac2a681c7_isr_1123954_33.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b84f33d8c0b7924829b850eb896a7af with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash c193793a22eede83d836c6ec76b07692 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a7ef0f07-780f-4fb3-ad14-d16c7f0f2e40_characterizeImage_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a7ef0f07-780f-4fb3-ad14-d16c7f0f2e40_characterizeImage_1123954_26.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/47732bfb-e82b-4b22-9bc0-f3d14903c229_isr_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/47732bfb-e82b-4b22-9bc0-f3d14903c229_isr_1123954_26.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c336dc39848eb4170d91c69017d56d with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash b37c1a64d9306da0b88b57e73adbec4a parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/16037a06-1ab0-46b8-8dec-51c557cd0566_characterizeImage_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/16037a06-1ab0-46b8-8dec-51c557cd0566_characterizeImage_1123954_20.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/f3a8d742-8069-4e0b-801f-b7673f18d51d_isr_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/f3a8d742-8069-4e0b-801f-b7673f18d51d_isr_1123954_20.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d162e79be483012897b0761f218d6f23 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 487f3bb7d6d1e9c23d93b3583c090732 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0b85f343-a708-4f8b-949d-f0ce503f0ef1_isr_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0b85f343-a708-4f8b-949d-f0ce503f0ef1_isr_1124087_50.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f338e66fc30e748ff9a777ab2b56d28 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 9e58bffff67bb791c13924d14b4ae5d5 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a89be1a6-681f-4da9-b6f3-5952a92855eb_isr_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/a89be1a6-681f-4da9-b6f3-5952a92855eb_isr_1124087_56.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69061c90e1b360980e3432669b1f3e16 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 82e0d799b2a45b49a74ea8ffe5627c88 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/eb102921-10c5-446d-a1bd-1b469d0f4268_characterizeImage_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/eb102921-10c5-446d-a1bd-1b469d0f4268_characterizeImage_1123954_41.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ff7dd953-92b2-48df-a16b-2685c9a8b359_isr_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ff7dd953-92b2-48df-a16b-2685c9a8b359_isr_1123954_41.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36775011e4bbe2fbccc401d095c0deaf with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 9fb0b9570e1a5e95aaf1ba6f37fe639d parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e028b4a1-7f80-4366-b05b-ed2f3829198a_characterizeImage_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e028b4a1-7f80-4366-b05b-ed2f3829198a_characterizeImage_1123954_51.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/452ebc2c-2b83-4017-8b54-e2d3c790f9c8_isr_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/452ebc2c-2b83-4017-8b54-e2d3c790f9c8_isr_1123954_51.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cee9b4b1c49c73a759d9128446c7d53 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash d17cb0893c545c8b45d1d23120a82db8 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f22306b3-49d2-47cc-916b-572f1eee7d9b_characterizeImage_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f22306b3-49d2-47cc-916b-572f1eee7d9b_characterizeImage_1123954_62.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/85643c29-9e15-42ab-8744-2b67a1f7959a_isr_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/85643c29-9e15-42ab-8744-2b67a1f7959a_isr_1123954_62.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59d073453987ec7f14e009b658ba3521 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 9ee496ae044b900e149a690bbd4b5824 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/371666f9-0f49-44fd-8a9a-1fcc99da5cd0_characterizeImage_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/371666f9-0f49-44fd-8a9a-1fcc99da5cd0_characterizeImage_1123958_53.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/52fb89d6-a739-4235-89ed-643373ddd13a_isr_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/52fb89d6-a739-4235-89ed-643373ddd13a_isr_1123958_53.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d21d0b667af227fbc67812208f34a81 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 72f2be2238c5d04d2aae9e9b935e067b parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f35c33a5-77d7-4da1-b9d3-2f1e33aa4d2a_characterizeImage_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f35c33a5-77d7-4da1-b9d3-2f1e33aa4d2a_characterizeImage_1123958_50.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a777548e-70a0-4a9d-bdc7-3846f50e28b9_isr_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/a777548e-70a0-4a9d-bdc7-3846f50e28b9_isr_1123958_50.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59df167262acc6bb30a6c5e5a1c81724 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash bb6f2a67c66028e158ccfec1bcec9f0f parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/186e0916-aef7-454a-8628-5672a18d8246_characterizeImage_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/186e0916-aef7-454a-8628-5672a18d8246_characterizeImage_1123958_62.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8287bd2c-b6bc-49b4-a8e2-1982c4261cd8_isr_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8287bd2c-b6bc-49b4-a8e2-1982c4261cd8_isr_1123958_62.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a59d334562d92dc1e1e5e2ac92d9427 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash dd4a8f92fa928307b9e41bd81bd2bdfe parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/da50ffbd-bc45-4ba4-aed1-e6f49b1635c0_isr_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/da50ffbd-bc45-4ba4-aed1-e6f49b1635c0_isr_1124087_14.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59f6224bc8417610fa41c1962460c3d7 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 9b11d8c391f1add22d2e21dd6bc55f26 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/7460d1dd-f2c2-455b-8117-6a67b8111011_isr_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/7460d1dd-f2c2-455b-8117-6a67b8111011_isr_1124087_30.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52fe4915a77bf106488636ca4f555416 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash e613144884e20f794e9499d4b1b3260a parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a3ec7e6b-505b-42ef-8eef-86406f1b677b_characterizeImage_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a3ec7e6b-505b-42ef-8eef-86406f1b677b_characterizeImage_1123954_18.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d64d8db7-0d44-4fd4-a16b-6729f4c1d40b_isr_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d64d8db7-0d44-4fd4-a16b-6729f4c1d40b_isr_1123954_18.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941283786a8815fdfcb1a5ddb1bd25bf with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash e0b74cba647a4e011d1a83963a3ce6b8 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fe8f6e1-f771-44c1-a894-41de2d7eb0c8_characterizeImage_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fe8f6e1-f771-44c1-a894-41de2d7eb0c8_characterizeImage_1123954_49.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9daa28d-012a-4bca-8b9e-7a97500a542b_isr_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/d9daa28d-012a-4bca-8b9e-7a97500a542b_isr_1123954_49.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28dcb4916e08637961d65da073fa5775 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 743828eba8e08f7a55bf571ac6cc0b92 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1294e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0a1f55c9-cb9d-407d-93fd-ddbed6aac68c_characterizeImage_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0a1f55c9-cb9d-407d-93fd-ddbed6aac68c_characterizeImage_1123954_60.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ae34244f-9ca5-4244-a09a-bf8a9b01f1f0_isr_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/ae34244f-9ca5-4244-a09a-bf8a9b01f1f0_isr_1123954_60.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 446d8e561c19c45cfc99cf50d47d7def with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 7fbd4600444e0334f01ea764e1119cce parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7a7880bc-ab1c-4a63-9027-bf3c02199de8_characterizeImage_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7a7880bc-ab1c-4a63-9027-bf3c02199de8_characterizeImage_1123954_9.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e3ae11d7-d974-4da1-99d6-5d2415e18570_isr_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e3ae11d7-d974-4da1-99d6-5d2415e18570_isr_1123954_9.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35ef14e0f004d40cc44f80cd949eeb83 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 316b40e60a7be8bddd087f21d116248a parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7b65691a-e011-4f03-a945-f552713e3fc4_characterizeImage_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7b65691a-e011-4f03-a945-f552713e3fc4_characterizeImage_1123958_4.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/28643950-37b0-4a4c-828f-8c88bee6f86a_isr_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/28643950-37b0-4a4c-828f-8c88bee6f86a_isr_1123958_4.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d097b9f09ef83517612e9a5b1ee1d263 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 71e391097c53ed91a2fb1d2652808654 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/afe6afee-437a-4513-a699-18c943ce7488_characterizeImage_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/afe6afee-437a-4513-a699-18c943ce7488_characterizeImage_1123954_55.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/22cd439c-5cc3-4d94-b0f0-b202f8785ded_isr_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/22cd439c-5cc3-4d94-b0f0-b202f8785ded_isr_1123954_55.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd122ecb55ac075b6575e3570e89c423 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash ad80eee1f4ce4d4cac7dabd676463a41 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/73123d8b-e271-4019-9827-b2eb03c02aa5_characterizeImage_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/73123d8b-e271-4019-9827-b2eb03c02aa5_characterizeImage_1123954_58.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e4fc7132-a22b-429d-8245-f49b5c476098_isr_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e4fc7132-a22b-429d-8245-f49b5c476098_isr_1123954_58.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52ba6f1c719b91263bfbeda1343968d0 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash f503c85b76021efdc5f2bcc713bdc08d parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f4794909-308b-4d16-af1d-b5c40ddc9b6e_isr_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f4794909-308b-4d16-af1d-b5c40ddc9b6e_isr_1124087_13.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62cd094c54a0ee8622e2ca4122f74d30 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 077a92be8651eb7013542b5d4fdebe42 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/aadd29df-e339-465a-a4cc-01b9fe70a798_isr_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/aadd29df-e339-465a-a4cc-01b9fe70a798_isr_1124087_16.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb91d9b15c712b4330a631292faa2eb with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 7bbf191a2c92df9540b1e43f7a147f5a parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07ff022d-e502-4777-8e19-8c0e0ddfeede_isr_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07ff022d-e502-4777-8e19-8c0e0ddfeede_isr_1124087_24.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82635080c0ce9bb72b90e6825e692344 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 47c0f12e0d3f4a51965309ec4c4b82af parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f71607de-d951-411c-8fd7-aa8d74466ce6_characterizeImage_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f71607de-d951-411c-8fd7-aa8d74466ce6_characterizeImage_1123958_26.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/315aca90-503e-4827-a75e-d0f68662ab15_isr_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/315aca90-503e-4827-a75e-d0f68662ab15_isr_1123958_26.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9728f5406d11f73ded3f8cc4552b241 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 7c3d9243c5562688e5dd92f6a16aa29b parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6f54847f-dc90-4e1b-919f-2c1991484186_characterizeImage_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6f54847f-dc90-4e1b-919f-2c1991484186_characterizeImage_1123956_58.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/18ef4a72-823b-4c05-8684-1d5e4de86aed_isr_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/18ef4a72-823b-4c05-8684-1d5e4de86aed_isr_1123956_58.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf1e2ef8921b14a4703143280fbbaf3 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 5e513f4c7c1d86843c654d44bdb47291 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/38ff1286-48b9-42b9-916c-202e7c3a1305_characterizeImage_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/38ff1286-48b9-42b9-916c-202e7c3a1305_characterizeImage_1123958_61.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/03cf367a-40d8-4486-8263-708b77ffce48_isr_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/03cf367a-40d8-4486-8263-708b77ffce48_isr_1123958_61.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94da4bc718d0188f9adb0c909e7c8a00 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash b3c3f1db9005c5ce5afe7440b1968f5f parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b771d235-4ee3-4721-b9c2-8a4b8f9221f4_characterizeImage_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b771d235-4ee3-4721-b9c2-8a4b8f9221f4_characterizeImage_1123958_42.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/73ac543a-9d03-4c0f-a9c0-66ffce927bf1_isr_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/73ac543a-9d03-4c0f-a9c0-66ffce927bf1_isr_1123958_42.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be39d3cd8653f74bd8927b5d9f75678f with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 653186249736e1e9b4602f84088c307b parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/812ca4b8-8327-4ffe-9450-1dce5c7c52cf_characterizeImage_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/812ca4b8-8327-4ffe-9450-1dce5c7c52cf_characterizeImage_1123958_48.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8df46b03-615f-407a-8f0b-ffd4b29e586f_isr_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8df46b03-615f-407a-8f0b-ffd4b29e586f_isr_1123958_48.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 019307c3a3dfe7724e5430d4a825af1f with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash d9f2ad97e42c07e5b760c2352a1f2433 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/384071a2-801a-44c6-a325-89037496a9bf_characterizeImage_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/384071a2-801a-44c6-a325-89037496a9bf_characterizeImage_1123958_14.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8b7c4646-136a-4771-a82d-abb51f207201_isr_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8b7c4646-136a-4771-a82d-abb51f207201_isr_1123958_14.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4b7767dbf9d036bc8400092a148cc4f with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 7e28b64007360f68cbc861ee13394355 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/44befd7c-93a1-4b6b-9487-993daf04a71e_isr_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/44befd7c-93a1-4b6b-9487-993daf04a71e_isr_1124087_60.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8709b6193a61497ae82c3c31f337bc03 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 6b412f28916c5a7107865ca67bfeb661 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/acf5759c-1d87-4987-a51e-c96c09bdf6cc_characterizeImage_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/acf5759c-1d87-4987-a51e-c96c09bdf6cc_characterizeImage_1123956_20.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec2f818a-4a34-4add-98d4-e93321450e92_isr_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec2f818a-4a34-4add-98d4-e93321450e92_isr_1123956_20.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68feb349f3c746714113d5d5570684ee with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 64e3b4fa6cb74e2874fa047dc81720fb parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b542ddc5-58d4-4abc-bc99-29ba2ae7ae60_characterizeImage_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b542ddc5-58d4-4abc-bc99-29ba2ae7ae60_characterizeImage_1123956_12.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7dda3b50-e575-49ec-8f1d-1b29e5991d04_isr_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7dda3b50-e575-49ec-8f1d-1b29e5991d04_isr_1123956_12.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3768843225fa2cc7cf05002ec927c36 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash cb2d29aa89135698a92ce79ef4b80787 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/047ccf90-259e-4689-a70e-493c0c51ae43_characterizeImage_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/047ccf90-259e-4689-a70e-493c0c51ae43_characterizeImage_1123958_52.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e795a5fe-63f2-4182-8aa5-d990dd741963_isr_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e795a5fe-63f2-4182-8aa5-d990dd741963_isr_1123958_52.stderr 2024-03-07 03:38:33 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:38:33 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:34 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:34 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:34 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7584268b0ef0f2cca5537355bb35bb with result from task 501 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:34 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash c84594cee11d6839a3da1a3596266118 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3813206f-9bd5-4946-985e-c3741133ccbd_characterizeImage_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3813206f-9bd5-4946-985e-c3741133ccbd_characterizeImage_1123954_15.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98c1821c-3d46-40fa-b1c4-e8b7a60f7d3f_isr_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98c1821c-3d46-40fa-b1c4-e8b7a60f7d3f_isr_1123954_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:34 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77374421f86fb6ab6f8827eed3f346d5 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 0b0f1ad1a11e37b5a16f45a961e35507 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d213ad62-b453-47e6-8001-9a217d76f0c8_characterizeImage_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d213ad62-b453-47e6-8001-9a217d76f0c8_characterizeImage_1123956_18.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3753fa64-a50e-4963-bdbc-b7defef899c1_isr_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3753fa64-a50e-4963-bdbc-b7defef899c1_isr_1123956_18.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d666fa808594046936a83f89fc2edf81 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash 6b6674ebcadc99ccee11511c2916929f parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/af8fb548-3425-4acb-9131-50b95a81c97f_characterizeImage_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/af8fb548-3425-4acb-9131-50b95a81c97f_characterizeImage_1123956_51.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/13fe12a9-2f42-44de-9d39-617b12595670_isr_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/13fe12a9-2f42-44de-9d39-617b12595670_isr_1123956_51.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956d6170d7864e40529f2cf3f7eda9ef with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 918bb1c25e966d3ba996ca329e594e1b parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/8dd71392-b0e4-4923-a96e-2bb9820dbacd_characterizeImage_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/8dd71392-b0e4-4923-a96e-2bb9820dbacd_characterizeImage_1123954_27.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dcf25a6f-a6cb-4ed2-8e12-9abaf0d5ae04_isr_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dcf25a6f-a6cb-4ed2-8e12-9abaf0d5ae04_isr_1123954_27.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 682823ada065ff011d04966679b9005f with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 83436b1bb1675282fef2ea3ec779392c parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07f1f4f9-08c2-498e-acb4-a5f4e8275023_isr_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/07f1f4f9-08c2-498e-acb4-a5f4e8275023_isr_1124087_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:34 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ee132f759887c2b34304931013d462e with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash d0d16f8a79cb1bab18197620f17f5ab8 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/790fa9e4-d5cc-44ae-b9de-882af8d655ac_characterizeImage_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/790fa9e4-d5cc-44ae-b9de-882af8d655ac_characterizeImage_1123956_1.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/f3070a0b-b96a-4f0d-bfaf-b68a7a11046d_isr_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/f3070a0b-b96a-4f0d-bfaf-b68a7a11046d_isr_1123956_1.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f936eb3caf2be492db486bf39e0d4d81 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 363b64fdab6918421123bfca0735f94a parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1716c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1ff549b3-c59b-4a6d-a203-4ded6e8a8997_isr_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1ff549b3-c59b-4a6d-a203-4ded6e8a8997_isr_1124087_48.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6b5230c1faa1e1429da480a567fc8f1 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 236315fc6aa79ee407b7dd3f3b6a92ce parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1720c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2b52c81e-4f35-4a7a-866c-a135ceef91e9_characterizeImage_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2b52c81e-4f35-4a7a-866c-a135ceef91e9_characterizeImage_1123954_5.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dffc42f3-8ca7-46f5-959c-5304f5b29560_isr_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/dffc42f3-8ca7-46f5-959c-5304f5b29560_isr_1123954_5.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d05e6b29db844e9f43e9be62eb2e97a with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash cd16c8c59b942bbe064c9eb2e7845f80 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e42a471-1c74-4c93-8f23-e2f5db1d8b70_characterizeImage_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e42a471-1c74-4c93-8f23-e2f5db1d8b70_characterizeImage_1123956_24.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/d154e555-4eda-41e6-961c-a37f3b869d34_isr_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/d154e555-4eda-41e6-961c-a37f3b869d34_isr_1123956_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:35 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3916f14471552e4f4062e7b50afff00a with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 86e793ca89fa88f4ab668734bbdfcbd3 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1115a916-728f-4eb8-875d-742548fc925c_characterizeImage_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1115a916-728f-4eb8-875d-742548fc925c_characterizeImage_1123954_57.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a3b3c7e7-d299-4dc0-8862-1b2be7016f44_isr_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a3b3c7e7-d299-4dc0-8862-1b2be7016f44_isr_1123954_57.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64ce5613223dfc83dd8b14e39c827017 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 47f4d4447ab1302d3ca6abd182923b02 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/6c48e18d-7626-4877-a962-e6fda789091e_isr_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/6c48e18d-7626-4877-a962-e6fda789091e_isr_1124087_17.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d42e458f33444690e2fbddefab8958d9 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 45cc0282848d7c3f11b3f77e8d5724f0 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/83425bc8-c9a6-4ca7-af7b-5ae323e82629_characterizeImage_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/83425bc8-c9a6-4ca7-af7b-5ae323e82629_characterizeImage_1123954_39.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/721d29eb-3cf7-4a0c-b081-1bba25bb0cec_isr_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/721d29eb-3cf7-4a0c-b081-1bba25bb0cec_isr_1123954_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:35 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1776ae8b3dd1f3459acf3182b8aab6e with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash c66aaadbf21907522ac40b342a1e59a8 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/d3e7a4e8-d9aa-4a5a-a926-01d444fd89b7_isr_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/d3e7a4e8-d9aa-4a5a-a926-01d444fd89b7_isr_1124087_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:35 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96935d6dd696e61adf86dd966e9bf386 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 6d2dc5b4d095d667e43a5912ba971723 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873242c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fed4baa6-4664-4291-8768-e2237c9682f1_characterizeImage_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fed4baa6-4664-4291-8768-e2237c9682f1_characterizeImage_1123954_25.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/924fb9f6-c471-4f09-87bb-40135b5204bd_isr_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/924fb9f6-c471-4f09-87bb-40135b5204bd_isr_1123954_25.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3ae1cc3f994746d98e08693bebf5012 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash bbaf790f2318605eb2edfcb2e01375ee parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1866feac-2038-44d4-89b4-da7c7dc92298_characterizeImage_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1866feac-2038-44d4-89b4-da7c7dc92298_characterizeImage_1123954_45.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/25365f10-e760-4c94-aa21-876bf56cb88b_isr_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/25365f10-e760-4c94-aa21-876bf56cb88b_isr_1123954_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:35 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aef54c419e4566206aefe4ed56da7f9 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 756e6f7ff78a8cf4e5ff4a1668a2911c parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83834772-88d8-4d01-82af-53beb33c78b8_characterizeImage_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83834772-88d8-4d01-82af-53beb33c78b8_characterizeImage_1123956_59.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/97db91c3-6edd-43ed-8066-9de5e6806d5f_isr_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/97db91c3-6edd-43ed-8066-9de5e6806d5f_isr_1123956_59.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 207bf81c68cfa3e45e8c491b26ac13a4 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash d1f8917c8f20024e36ad4c00c5581a71 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/f21f079d-efdc-41db-a473-72895056d33d_characterizeImage_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/f21f079d-efdc-41db-a473-72895056d33d_characterizeImage_1123956_49.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2feb6b4f-f0c9-44c2-b9b0-348c5b171774_isr_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/2feb6b4f-f0c9-44c2-b9b0-348c5b171774_isr_1123956_49.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f0c1df8bd762c9c2aa3d47d5c801b86 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash d4212e9f97d2818b9b2966fd4489612d parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2385db8e-8dab-458c-a821-c26c96228b4d_characterizeImage_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2385db8e-8dab-458c-a821-c26c96228b4d_characterizeImage_1123954_38.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/3a082f0f-c861-4712-a38d-4585eae9d424_isr_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/3a082f0f-c861-4712-a38d-4585eae9d424_isr_1123954_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:36 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9800e24b34638fe7518f8fd682793cc with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash eae5378f086b2cd1be9bd9c709ee6bd8 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/5f7d0e68-f6c5-46b8-b1cb-f6e8f6d8358a_characterizeImage_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/5f7d0e68-f6c5-46b8-b1cb-f6e8f6d8358a_characterizeImage_1123954_53.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfce2375-9a62-41c0-8d9b-75b89acff708_isr_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bfce2375-9a62-41c0-8d9b-75b89acff708_isr_1123954_53.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70a1766168f7634c84a744ccd8019227 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 8f40169e9adde144ad582596f9dbb4e5 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/476f4296-62c3-4c9d-b32d-b3fc7e421357_characterizeImage_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/476f4296-62c3-4c9d-b32d-b3fc7e421357_characterizeImage_1123954_4.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bb4c774c-1d28-4b66-b9f2-6926cd5b8a07_isr_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/bb4c774c-1d28-4b66-b9f2-6926cd5b8a07_isr_1123954_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:36 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89b48f79130b8df724d256e5d4482c9c with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 45d6a8ba54f5e6ee2c28eb1095b94851 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1737e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3e80bdd1-c28b-4e3e-930c-1c376a984388_isr_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3e80bdd1-c28b-4e3e-930c-1c376a984388_isr_1124087_61.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8bd283e18e237adfa95e433c5e544c0 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 50a718b03053643f427761329f61e9e7 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873251c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 411 parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b51d1054-8efb-4dc0-bf74-c1a18341bee8_characterizeImage_1123956_55.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b51d1054-8efb-4dc0-bf74-c1a18341bee8_characterizeImage_1123956_55.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ce1e0181-b065-488a-bd24-5c8e5f0c410e_isr_1123956_55.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ce1e0181-b065-488a-bd24-5c8e5f0c410e_isr_1123956_55.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 921a42e398c86bbc77425a5da034c28a with result from task 510 parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:36 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 4461331b207912c314c88448f3074791 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8263b61f-4011-4957-abc8-3af7883f9ada_characterizeImage_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8263b61f-4011-4957-abc8-3af7883f9ada_characterizeImage_1123958_43.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/34afe697-0f70-4a66-9cc2-945b667bc208_isr_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/34afe697-0f70-4a66-9cc2-945b667bc208_isr_1123958_43.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9863495f1c84aa1fe8828ce666cace44 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash f170f569dee718b14ca2f3e5b8e9f581 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1727a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 413 parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0eca7db8-376d-4193-823a-d30412165370_characterizeImage_1123956_36.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0eca7db8-376d-4193-823a-d30412165370_characterizeImage_1123956_36.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c4352610-e962-48b7-b464-305eac7c1775_isr_1123956_36.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/c4352610-e962-48b7-b464-305eac7c1775_isr_1123956_36.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry bc88ab4dd86c2534d609161cd490e56d with result from task 528 parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:37 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 8970d5e191d44c57151af1768dd92906 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b33012aa-aa26-4c3f-98f8-141afadb4352_isr_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b33012aa-aa26-4c3f-98f8-141afadb4352_isr_1124087_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:37 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:37 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:37 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:38 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:38 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:38 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:39 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:39 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:39 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6c52bfe816d7ed7df71b8446aa87e3f with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 3fb2648f636aa96f82dd546aee16de2e parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b06d124b-bc58-46f0-8c0a-f05d0a17364c_isr_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b06d124b-bc58-46f0-8c0a-f05d0a17364c_isr_1124087_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:40 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:40 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 109347692e688335e3bddaf22bc02bcc with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 8180c94b18f675755141d64ec52d82a4 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/348d183f-0c66-4e3e-8b48-09aa22d1cc23_characterizeImage_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/348d183f-0c66-4e3e-8b48-09aa22d1cc23_characterizeImage_1123956_53.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/33c39191-92f2-476e-963a-d2784666aaf3_isr_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/33c39191-92f2-476e-963a-d2784666aaf3_isr_1123956_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:40 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34bb5b09cadb2c43eb829c4f150c7bbc with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 5e196acf326a5f34a6381ef10c3f3581 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3fca6a36-a2c3-41db-99fe-13587a4b605f_isr_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/3fca6a36-a2c3-41db-99fe-13587a4b605f_isr_1123958_2.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2418809a2400448e3ff7a5ee7e5aaa5b with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash ce6b71be7acce86cecfcca01dde72da0 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e1be5409-c81d-4160-b560-fad915cc0b04_isr_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/e1be5409-c81d-4160-b560-fad915cc0b04_isr_1123954_2.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35f35a7b3cea950836108d2785601086 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 09c18a2a96187721ccd69fe9bbc484a7 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/98cf117a-5151-415f-b174-9c48dd8c31c2_characterizeImage_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/98cf117a-5151-415f-b174-9c48dd8c31c2_characterizeImage_1123954_28.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1bed2145-3b2e-4a4c-9082-17a2709f83c9_isr_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/1bed2145-3b2e-4a4c-9082-17a2709f83c9_isr_1123954_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:41 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41b4a59ee659ee8fba4570b03bbcc810 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash e70676d234d2e724a0549b8244a8ef97 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/930d4d3f-af55-435e-b76a-cc9da56844e9_characterizeImage_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/930d4d3f-af55-435e-b76a-cc9da56844e9_characterizeImage_1123956_43.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a0b08c4c-2fac-49fe-bd64-ecd03ba08ecf_isr_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a0b08c4c-2fac-49fe-bd64-ecd03ba08ecf_isr_1123956_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:41 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ade245d754136e38253501982c8bc560 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 6cdbfce54988d152a0bd842ddb511bd1 parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/17828148-447b-49bd-8197-9a4c9a11159b_characterizeImage_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/17828148-447b-49bd-8197-9a4c9a11159b_characterizeImage_1123956_50.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/717bde92-c332-4576-a4b0-5ab4d408cad9_isr_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/717bde92-c332-4576-a4b0-5ab4d408cad9_isr_1123956_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:41 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:42 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30d4a745d6972c89d176f8cea328f64b with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash f2a221d9b713d7c341950195155dfa6e parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7df1f56a-1848-4d9e-86b5-06c1f0bf7ea4_characterizeImage_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7df1f56a-1848-4d9e-86b5-06c1f0bf7ea4_characterizeImage_1123954_22.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8a1c379e-9d9e-41bc-90d9-4a2089a4007e_isr_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/8a1c379e-9d9e-41bc-90d9-4a2089a4007e_isr_1123954_22.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed8e98bf6c53b7175f0d41b63c8f2a8 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 5e64ff3c30c2c91e344ac17165031b7f parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cca48b58-5998-4ffa-9c58-1471409a22f2_characterizeImage_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cca48b58-5998-4ffa-9c58-1471409a22f2_characterizeImage_1123958_19.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/083acf69-2a2a-424f-a8a6-db438c28cb9d_isr_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/083acf69-2a2a-424f-a8a6-db438c28cb9d_isr_1123958_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:42 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5022068ddaa9eae2294b7c81ab1dcd00 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash c80a4c4839afccc6d908b42174a96921 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47d7bb4c-757e-4461-a69d-8d5605ebc28b_characterizeImage_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47d7bb4c-757e-4461-a69d-8d5605ebc28b_characterizeImage_1123956_61.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dc979ffe-ec1e-489b-b6d6-a486cba6212c_isr_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/dc979ffe-ec1e-489b-b6d6-a486cba6212c_isr_1123956_61.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f8ac01b6e777c4a3c8c45ab358c4014 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash a6197411640117f0ff3492f7c2ab81c5 parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1c3d480e-ee30-4d31-a94c-85af7d845958_characterizeImage_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1c3d480e-ee30-4d31-a94c-85af7d845958_characterizeImage_1123956_6.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/99356537-0e99-47ac-bdc7-77335393996a_isr_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/99356537-0e99-47ac-bdc7-77335393996a_isr_1123956_6.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 671253f2db013bdf26ec78b29fc1cd8c with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 38b1d10cdf915eec3e4b426d386fcd1f parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f4296afa-5cea-4841-bc92-b38f58a0b429_characterizeImage_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f4296afa-5cea-4841-bc92-b38f58a0b429_characterizeImage_1123954_6.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/300d0f50-48fa-4af3-a143-29fc5d09f74f_isr_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/300d0f50-48fa-4af3-a143-29fc5d09f74f_isr_1123954_6.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acc58f1b625748ee7b8b73a181003ad8 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 24b5f9a1817d60ccca15f2d84dd4de95 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a7f8db4-0bc2-4a7d-b80e-870a03f82aa4_isr_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0a7f8db4-0bc2-4a7d-b80e-870a03f82aa4_isr_1124087_49.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7be4b55c943168027b32aaf9e8cd2556 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 6be2c6d8428251f30e2527a91e8fe845 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/926ee7f2-a424-4f11-b7d0-2b8bffa03f35_characterizeImage_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/926ee7f2-a424-4f11-b7d0-2b8bffa03f35_characterizeImage_1123954_44.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6fe81655-9f68-4835-adc4-93a3a7fcace8_isr_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6fe81655-9f68-4835-adc4-93a3a7fcace8_isr_1123954_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:42 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00f6a65d5e21a7b760f7270e17c82b33 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 758283cbaa3d600bae5289a3aeea992d parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ea566331-cf92-43e9-b607-7059131152dc_characterizeImage_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ea566331-cf92-43e9-b607-7059131152dc_characterizeImage_1123958_32.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7564522-09be-489d-b8b7-d2681b1b31ec_isr_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e7564522-09be-489d-b8b7-d2681b1b31ec_isr_1123958_32.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610b4bb6b9239c22567c0f655f96bafc with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 23a5be7c5d04de8bfa4fa4662a8ca6e8 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7283b45e-9eae-4036-b98b-d90206f5e883_characterizeImage_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7283b45e-9eae-4036-b98b-d90206f5e883_characterizeImage_1123956_16.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/85cc6439-692e-43a5-877b-e834843fe89c_isr_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/85cc6439-692e-43a5-877b-e834843fe89c_isr_1123956_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:42 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce61961ee50c1e8d7e78058e8e1c8e96 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 64f5748f4932d4a5db24e745cb024211 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d4fd0b0e-c4ef-4fea-ae72-ad782d130769_characterizeImage_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d4fd0b0e-c4ef-4fea-ae72-ad782d130769_characterizeImage_1123956_39.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8da616e1-725f-43af-b32e-9e57e19c0838_isr_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8da616e1-725f-43af-b32e-9e57e19c0838_isr_1123956_39.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f16d664b17fa9a475afb03896adabbaa with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 963718ed5a97b3a561d403e9196ac271 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/bb9d4330-ebea-47ab-88e8-6698eff01bb0_characterizeImage_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/bb9d4330-ebea-47ab-88e8-6698eff01bb0_characterizeImage_1123956_48.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b7a4f39c-cd5f-47f2-85b8-0f7d7187c709_isr_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b7a4f39c-cd5f-47f2-85b8-0f7d7187c709_isr_1123956_48.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b3d289480f5ebd6a6f8ad398cde2c7a with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 15e2b6d411ee96194d43bafeb8a46a56 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/82812d42-4be7-4fdb-8f31-ceebbfae4060_isr_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/82812d42-4be7-4fdb-8f31-ceebbfae4060_isr_1124087_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:42 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3eb5ed2b4d0741cb30443cbd089b65f with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 459a13d380f12fdc88ad545b5459f70f parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/be5fa37a-5e9a-466f-b147-22da03eb56ab_characterizeImage_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/be5fa37a-5e9a-466f-b147-22da03eb56ab_characterizeImage_1123954_24.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b671a302-f1f7-4d2e-a0f9-58d4c7678ca2_isr_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b671a302-f1f7-4d2e-a0f9-58d4c7678ca2_isr_1123954_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:43 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63765065ef40ab496509fd5a7a999c84 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 41858345246c09e368d8662a258c5c19 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3c6c2c9f-e427-4c17-a2b8-31529977ce0b_characterizeImage_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3c6c2c9f-e427-4c17-a2b8-31529977ce0b_characterizeImage_1123954_12.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a263dd41-9085-4a08-8bc5-19a3aefb4e7c_isr_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a263dd41-9085-4a08-8bc5-19a3aefb4e7c_isr_1123954_12.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2b106a3534b725afcc3edce088c83cb with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 16fdf632038b58740a74668b6c9d0824 parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/266c88b2-1b04-4047-be8f-e747dd171401_isr_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/266c88b2-1b04-4047-be8f-e747dd171401_isr_1124087_45.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd516d99cc07b2b5f268c14f5bc4c591 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 46054bda177d3fba5c4f772aac4763fa parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ea917666-fd94-4d21-b963-a3193548b1da_characterizeImage_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ea917666-fd94-4d21-b963-a3193548b1da_characterizeImage_1123956_23.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/590c566e-aeb6-44c4-a18e-a8c135e160f1_isr_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/590c566e-aeb6-44c4-a18e-a8c135e160f1_isr_1123956_23.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc8e7f3cae018a31ce958f5356cda1a with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 0755a13a0b3a9c1581f38e73d6222198 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/165a30ea-056f-4d38-ae0e-d7a592f3151d_isr_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/165a30ea-056f-4d38-ae0e-d7a592f3151d_isr_1124087_9.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b4b454a930106673dad01262e2c7893 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 546d9fbdd23feec01b2449ccb2579898 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a19b023c-349d-4113-b897-4e66c2c9ee01_characterizeImage_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a19b023c-349d-4113-b897-4e66c2c9ee01_characterizeImage_1123956_5.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/283b1c8b-bf1d-4445-b9f8-1f915fd389cf_isr_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/283b1c8b-bf1d-4445-b9f8-1f915fd389cf_isr_1123956_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:43 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ece3d4d0554d5470ff3eb4eb8f3626e7 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash b122bca9b780c7c85115f6bd323bc64a parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/546ef7bd-7e28-432c-add2-2cb9bb0b4c4e_characterizeImage_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/546ef7bd-7e28-432c-add2-2cb9bb0b4c4e_characterizeImage_1123956_33.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/40f2305f-316a-4745-adce-701d571f6508_isr_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/40f2305f-316a-4745-adce-701d571f6508_isr_1123956_33.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d51dc82498b2afdc42613f32d27359 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 4c16f607ceeaaebb98cd097ddaa67524 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/fdd6a21e-23c3-47f8-9b4a-cf3dea418dda_characterizeImage_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/fdd6a21e-23c3-47f8-9b4a-cf3dea418dda_characterizeImage_1123956_28.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ae4ad2c5-24a1-4648-b669-850fc8a673f5_isr_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ae4ad2c5-24a1-4648-b669-850fc8a673f5_isr_1123956_28.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dbe16b3332fed47c42a95eed09e40ed with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash f0f399c40fcb3910e2bafb7f71b9e285 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/17433f0a-fbd6-450f-850f-05c735643d89_isr_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/17433f0a-fbd6-450f-850f-05c735643d89_isr_1124087_10.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d67addd86c6797badf1f16072b7b7b82 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 196d136254572d1065715edbf8821ef2 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b845de0c-0aa1-4d06-98be-5bb63e430e64_isr_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/b845de0c-0aa1-4d06-98be-5bb63e430e64_isr_1124087_59.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59a150689cd108c0cab67c744de149e4 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash ecfd50c7ba4321186e68d885e4391cd3 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c37daf8c-0590-48f9-8233-329770484eb8_isr_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/c37daf8c-0590-48f9-8233-329770484eb8_isr_1124087_31.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16037f19d7fa742f2945132de45a0354 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash a9c00af72112cd6ea695962e424c1a54 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b9ff879a-1759-4b29-8b51-da8b13f22e6c_characterizeImage_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b9ff879a-1759-4b29-8b51-da8b13f22e6c_characterizeImage_1123958_40.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4d6dab0f-6191-43e2-8192-4b725418c82e_isr_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4d6dab0f-6191-43e2-8192-4b725418c82e_isr_1123958_40.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7c77b9bcde0e1cef2d9c8001ba89883 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 6648a3efe0e031fc24544e7786ef65cc parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3a0a9e2d-7da7-4539-b1ad-c0b77cd0b3f9_isr_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/3a0a9e2d-7da7-4539-b1ad-c0b77cd0b3f9_isr_1124087_15.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7359c7abdf54dfb7760cf707528c1cb7 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash e6042d6c1d3fbf4e7bb5f8287469354c parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1b65d045-50b0-4efe-bc2b-6ffef52f5b6a_characterizeImage_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1b65d045-50b0-4efe-bc2b-6ffef52f5b6a_characterizeImage_1123956_22.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a2ae7be4-7687-423d-b1ee-a3eba746b1c1_isr_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/a2ae7be4-7687-423d-b1ee-a3eba746b1c1_isr_1123956_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:43 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f390a1deb14ab211e74e7eab721686e3 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 79c3b04909a45d6b3670ee6885be7d60 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6dec1e93-25db-4189-a884-203088701fe0_characterizeImage_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6dec1e93-25db-4189-a884-203088701fe0_characterizeImage_1123956_44.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e0ce240e-0758-47d6-938e-bfb17102851f_isr_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e0ce240e-0758-47d6-938e-bfb17102851f_isr_1123956_44.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97d41bd2d351c9cc6bba31cf9d0e3477 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 38e1729962ccc5d0a83f49939d36f56d parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/782c6997-0b79-437d-9425-314e2a3bcfeb_characterizeImage_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/782c6997-0b79-437d-9425-314e2a3bcfeb_characterizeImage_1123958_41.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9360a82b-58e0-4fdd-a4eb-0dfea60807ff_isr_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/9360a82b-58e0-4fdd-a4eb-0dfea60807ff_isr_1123958_41.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd77d0922baba32f736b01bb64b6cfb with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 5f048ec3f471c2e4f50e5dcb84b29045 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1b4db1c8-e075-4771-a702-dabc1a1a382c_isr_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/1b4db1c8-e075-4771-a702-dabc1a1a382c_isr_1124087_46.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b0c7a11ff76d8b41910312f8115cf9b with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash b7510edeadcbac71713f0550a200eb97 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e56b7148-cdc2-4a43-9758-72f6f75d1610_characterizeImage_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e56b7148-cdc2-4a43-9758-72f6f75d1610_characterizeImage_1123954_34.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/0b7b76a9-a74b-4a46-952e-7fde7345c9a3_isr_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/0b7b76a9-a74b-4a46-952e-7fde7345c9a3_isr_1123954_34.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21928088fb0f26fe3c464105c226bb6d with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 56828a78fd22e660a2e5abe48b10d708 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/12dce86b-1ccc-4bee-99ba-c6c59a6938e4_characterizeImage_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/12dce86b-1ccc-4bee-99ba-c6c59a6938e4_characterizeImage_1123958_55.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/75861e7f-4dd2-454a-a65f-b9f72228e364_isr_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/75861e7f-4dd2-454a-a65f-b9f72228e364_isr_1123958_55.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d48a5a24c75bb73596c2a3d154653be1 with result from task 633 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:44 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 8c01704185ff9f50f7e262c655b41039 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/80cbb4a6-4513-499f-9a46-5aecbc68b98e_characterizeImage_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/80cbb4a6-4513-499f-9a46-5aecbc68b98e_characterizeImage_1123954_10.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/598291ed-5525-40e0-8389-694a9b684fbc_isr_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/598291ed-5525-40e0-8389-694a9b684fbc_isr_1123954_10.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d012051f473313613c22e4a2857a0677 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 1e713d09849d31888595fbba3df09f82 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d5d926aa-06db-498a-af81-2a2f86d44cb4_characterizeImage_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d5d926aa-06db-498a-af81-2a2f86d44cb4_characterizeImage_1123958_16.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/6e85d617-0995-48ea-990c-3482d7d8576d_isr_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/6e85d617-0995-48ea-990c-3482d7d8576d_isr_1123958_16.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 859f3acff31733d7815ed6940b930fdd with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 6aab8db812c239accb75671a14ce495f parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c35bca51-eec9-40de-95c4-a229c0f56a5f_characterizeImage_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c35bca51-eec9-40de-95c4-a229c0f56a5f_characterizeImage_1123958_46.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/16e224d3-1269-42e5-8919-2e6304c06832_isr_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/16e224d3-1269-42e5-8919-2e6304c06832_isr_1123958_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:44 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bd238e42a70a5f7b39f3cbd4212f702 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash a203cd63add2c8a45d6d51685d1283a9 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/02b96a78-dbca-4b25-a31a-49cd0ff6a606_isr_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/02b96a78-dbca-4b25-a31a-49cd0ff6a606_isr_1124087_26.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96f09fd278915d06e77aeeb1d1b82b95 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 548ebecb202cf1d46aeb628e0b332b13 parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468872184a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/4f80d198-8ed1-4123-a75d-6dd1b72166fa_characterizeImage_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/4f80d198-8ed1-4123-a75d-6dd1b72166fa_characterizeImage_1123956_34.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b0f841c1-5ea4-43d8-84fe-b78c307340ab_isr_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/b0f841c1-5ea4-43d8-84fe-b78c307340ab_isr_1123956_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:44 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c317a812bba5d2990cdae92f253a4c09 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 9b25045ae5b1937f3efdf07aebe629c8 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7d97295a-da9f-446f-b72e-8d1fa7679303_characterizeImage_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7d97295a-da9f-446f-b72e-8d1fa7679303_characterizeImage_1123954_35.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98a6636a-20b6-4a94-8417-dccdd3e7497b_isr_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/98a6636a-20b6-4a94-8417-dccdd3e7497b_isr_1123954_35.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3588fbc93366f6b9ccbfd9827a47cf05 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash af00cd310382dda5b192d46dfb10d433 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e5cebe92-bb9c-42a2-bd49-5689981d104a_characterizeImage_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e5cebe92-bb9c-42a2-bd49-5689981d104a_characterizeImage_1123958_35.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1323fb68-b2b6-4987-ba62-cd9a2b848377_isr_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/1323fb68-b2b6-4987-ba62-cd9a2b848377_isr_1123958_35.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f68fa866a529562417c936d934be3021 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 36ebb9d3750d315256bcddbf119e1e25 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/98d55b79-8a9a-467d-a0d3-6969886b85f3_characterizeImage_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/98d55b79-8a9a-467d-a0d3-6969886b85f3_characterizeImage_1123956_47.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/19ecc46a-0bc9-4fb4-b971-493839d1595d_isr_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/19ecc46a-0bc9-4fb4-b971-493839d1595d_isr_1123956_47.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1a4c6ec18d6537d42c6a88b16912a07 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash fef4da965a2c2c070ce498aa9f18e652 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/cf7d6529-c6d7-43b9-b66d-f6fe104b22ca_characterizeImage_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/cf7d6529-c6d7-43b9-b66d-f6fe104b22ca_characterizeImage_1123956_60.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3c1bcd56-7d78-49f4-96cd-4ba3ac09f44c_isr_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/3c1bcd56-7d78-49f4-96cd-4ba3ac09f44c_isr_1123956_60.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfe6cd75394320a84cb6061654d4235f with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 1ab5d35a06af08e900ea87e9e803aeec parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/87bca6a0-35fa-4e7e-b483-f42a45f8fb1b_characterizeImage_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/87bca6a0-35fa-4e7e-b483-f42a45f8fb1b_characterizeImage_1123954_54.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/037ac91f-0bb8-4af8-8c42-582203dd0c3e_isr_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/037ac91f-0bb8-4af8-8c42-582203dd0c3e_isr_1123954_54.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d550bb9224029f4b48f63afa47710cc with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash f114c20ac0b64ae0ef146a01e003f477 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bc65d877-52c4-4a05-83a0-3ad7b188eb91_isr_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/bc65d877-52c4-4a05-83a0-3ad7b188eb91_isr_1124087_1.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef94093cb4853c7b408e7123e6f662d4 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 267d7096696584169f2027f5508cfb49 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a778b1e4-731f-4273-a848-7c61d5061912_characterizeImage_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a778b1e4-731f-4273-a848-7c61d5061912_characterizeImage_1123954_32.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/08df6f8c-d953-43d5-b1c7-ab42dcce212c_isr_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/08df6f8c-d953-43d5-b1c7-ab42dcce212c_isr_1123954_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:45 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:45 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36c508e4b86d4bb672407c6976a4726a with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 09a1934f9a61d193b695598aaf442c32 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96f93c18-b218-49b4-b63c-05074775da31_characterizeImage_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96f93c18-b218-49b4-b63c-05074775da31_characterizeImage_1123956_42.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e5877ca5-7337-4ea6-a52d-952ea7d079b8_isr_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/e5877ca5-7337-4ea6-a52d-952ea7d079b8_isr_1123956_42.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f915088e6ca7c9337815b8d4e3b2f0f2 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash a7007ec0a6812f3aab3655f69b526cb6 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873254e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/915ba7b6-04e8-4443-90fa-740470cd6b60_isr_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/915ba7b6-04e8-4443-90fa-740470cd6b60_isr_1124087_62.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 884df25ba0385bc5d0141d8c7e83a1ab with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash f469e8774b3789501d41763b9a7edae3 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8e867797-5eda-4eba-b69a-39258ff4d317_characterizeImage_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8e867797-5eda-4eba-b69a-39258ff4d317_characterizeImage_1123958_60.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4ff7e57-87ed-41f2-bfae-d47cd6713ea6_isr_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/e4ff7e57-87ed-41f2-bfae-d47cd6713ea6_isr_1123958_60.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db86ad77e0cbbbde5654cf9276b0c9ef with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash eacfe80d34dee1131af6ce38b1c1e0fd parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e6d96f7c-d0ec-4e7b-818e-ebe30d5517e2_characterizeImage_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e6d96f7c-d0ec-4e7b-818e-ebe30d5517e2_characterizeImage_1123958_47.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c7991d7-2bc2-4871-ae52-1f6757096c1c_isr_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/4c7991d7-2bc2-4871-ae52-1f6757096c1c_isr_1123958_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:45 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2e9c9a813c0ddf2e12de2aecdd4db5f with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash fc4568618e4a2c79e856d29f2d96c383 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c223182f-99a0-4aed-a159-a108ccf6028a_characterizeImage_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c223182f-99a0-4aed-a159-a108ccf6028a_characterizeImage_1123956_62.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7772a733-4f14-479b-9cb4-123cbc96b262_isr_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/7772a733-4f14-479b-9cb4-123cbc96b262_isr_1123956_62.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c77fa41ab1764794277bbb394fe21bd with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash efc29694e665941ad5a06c6a54239df9 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47a23f93-9f45-47b8-af63-2c1cbd78f7af_characterizeImage_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47a23f93-9f45-47b8-af63-2c1cbd78f7af_characterizeImage_1123956_56.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec0478e6-d38e-426f-82cb-2f98f9f70352_isr_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/ec0478e6-d38e-426f-82cb-2f98f9f70352_isr_1123956_56.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e3d1d3d88b56e26933e862200513f2b with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash e3373ec63ae88d27bd298db3eea7dd1d parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468872187c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e1f8447-0f54-448b-8e32-2643b854c7b1_characterizeImage_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e1f8447-0f54-448b-8e32-2643b854c7b1_characterizeImage_1123956_13.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/0ac1a99b-94a8-4c80-900a-8d775c98eb9b_isr_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/0ac1a99b-94a8-4c80-900a-8d775c98eb9b_isr_1123956_13.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90821d35a30c5d92c6cb2baf77b2b237 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 719af86669e2b5ad28901fca11c249ec parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5abb4c45-e01b-46b4-8a41-e926aa9a814c_isr_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/5abb4c45-e01b-46b4-8a41-e926aa9a814c_isr_1124087_22.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a58ed0a53fbd72854a6bb72d46a49fbd with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash a62c91331f448abc9081d6c1acc3eb33 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8930fe04-c4d0-47eb-8be5-8f68c72e7a1e_characterizeImage_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8930fe04-c4d0-47eb-8be5-8f68c72e7a1e_characterizeImage_1123956_9.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/84e04b2b-00a3-4f55-b368-f771c7d2fc4c_isr_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/84e04b2b-00a3-4f55-b368-f771c7d2fc4c_isr_1123956_9.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c40756aebd6f0505e68a24ac3b302424 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 7aab8781860c1a80161be6faf15d8e10 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/96b34a2b-abe9-46e1-8c59-ecca997b65fc_characterizeImage_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/96b34a2b-abe9-46e1-8c59-ecca997b65fc_characterizeImage_1123954_14.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a36aaed2-9357-4abf-ab47-2f4d0ce87966_isr_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/a36aaed2-9357-4abf-ab47-2f4d0ce87966_isr_1123954_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:45 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82741a32be1f54483047d5758a9f419d with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 02faf2d0109f697d03a513da10f20cb0 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/675b26ec-72f7-45e6-b143-f6e9a216ced7_characterizeImage_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/675b26ec-72f7-45e6-b143-f6e9a216ced7_characterizeImage_1123958_59.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8941d282-774c-425f-b0e1-ae15b3964492_isr_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8941d282-774c-425f-b0e1-ae15b3964492_isr_1123958_59.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb919814f5d8003d9334c1ddc42c51c8 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash a6b612697bef76f74dadc87a271f93a4 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/cdc98367-e89d-4f61-a6b0-639bfeec7c9e_isr_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/cdc98367-e89d-4f61-a6b0-639bfeec7c9e_isr_1124087_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:46 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f661dea83cd43627b57da1aed24fdc39 with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash aea96f0019e8ff0167c8db43d6329aca parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3393ebd2-9787-4b7f-81c7-6aa5b7e56868_characterizeImage_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3393ebd2-9787-4b7f-81c7-6aa5b7e56868_characterizeImage_1123958_29.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/33fd80e5-432c-454d-9154-fe5db47f8ca6_isr_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/33fd80e5-432c-454d-9154-fe5db47f8ca6_isr_1123958_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:46 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:46 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 316c4fc13742d2091d43ce995cccf716 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 816b0894c856ee7774b435e614e016d4 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1cdc9e05-ab4c-4e96-88b4-702eb1b7beac_characterizeImage_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1cdc9e05-ab4c-4e96-88b4-702eb1b7beac_characterizeImage_1123954_59.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9b26dc33-e841-45e5-a793-efd2dfa10855_isr_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/9b26dc33-e841-45e5-a793-efd2dfa10855_isr_1123954_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:47 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:47 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a994484d4ac8e269a13615252820f100 with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 75fc9247c1a6d01a7f76216ef5dae977 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7626e02d-eb24-48bf-b2e3-546c89f2f0ec_characterizeImage_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7626e02d-eb24-48bf-b2e3-546c89f2f0ec_characterizeImage_1123958_18.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/900ce0db-501b-40b6-a2cd-a1b7c7ced1ef_isr_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/900ce0db-501b-40b6-a2cd-a1b7c7ced1ef_isr_1123958_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:47 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:47 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f511da540448cf7812a0925cc32fd6e1 with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash b53c51e9ec9096b3e8ea3932dde314f9 parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3623f8cf-3826-45ed-99d3-27587c1eae06_characterizeImage_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3623f8cf-3826-45ed-99d3-27587c1eae06_characterizeImage_1123954_37.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/830f99cd-a19d-4ec8-858e-128efc8a291f_isr_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/830f99cd-a19d-4ec8-858e-128efc8a291f_isr_1123954_37.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4216aabc98c21d5b53c3c08824c54b18 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 41b81a8a366e8b489d6e99215b53924f parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688721a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/09333de0-8ee2-4517-81b8-b34948bceea4_characterizeImage_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/09333de0-8ee2-4517-81b8-b34948bceea4_characterizeImage_1123956_54.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8c9ea65b-7ed8-43f9-92da-af52325ef001_isr_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/8c9ea65b-7ed8-43f9-92da-af52325ef001_isr_1123956_54.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 697d7df3bbc0f40e7d8f8eb28f5607bb with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash fa505f78a961b2d889ac58cdf8197c1e parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a9ee0ca1-ca0a-431d-b3ac-0947c308f5ed_characterizeImage_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a9ee0ca1-ca0a-431d-b3ac-0947c308f5ed_characterizeImage_1123958_38.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b8d14d61-efef-4aaf-9859-7b9eeee838f7_isr_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/b8d14d61-efef-4aaf-9859-7b9eeee838f7_isr_1123958_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:47 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:47 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:48 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:48 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:48 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:48 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:48 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:48 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:38:49 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 223 try 0 failed parsl.dataflow.dflow INFO: Task 223 marked for retry parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 4cda3dff1034ed6eb3cdd5f0bf7e8a47 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 1 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 217 try 0 failed parsl.dataflow.dflow INFO: Task 217 marked for retry parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash cf93be5b5f6be3415f03e3ce320e0f96 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 1 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 0 failed parsl.dataflow.dflow INFO: Task 34 marked for retry parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash e89bad3fbb7a2001293003b481e85847 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 1 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 340 try 0 failed parsl.dataflow.dflow INFO: Task 340 marked for retry parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash a515cdccae2b9ce0ea2a92a449e4b3cc parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 1 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 19 try 0 failed parsl.dataflow.dflow INFO: Task 19 marked for retry parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 79660152977a0c14df444f7ebb7cad75 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 1 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 0 failed parsl.dataflow.dflow INFO: Task 52 marked for retry parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash ad6166e25c076303099816d1d40bb95c parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca551c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 1 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 0 failed parsl.dataflow.dflow INFO: Task 28 marked for retry parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 07e63588df970413f34c38152add60dd parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 1 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 220 try 0 failed parsl.dataflow.dflow INFO: Task 220 marked for retry parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash bd845f52772843c76417c5a384d2ccc0 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 1 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 226 try 0 failed parsl.dataflow.dflow INFO: Task 226 marked for retry parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash a52f7f454350b07705c818f2aadbd17d parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 1 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 256 try 0 failed parsl.dataflow.dflow INFO: Task 256 marked for retry parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash d556701645b0eb4bcbfa87cdf609571c parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 1 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 0 failed parsl.dataflow.dflow INFO: Task 40 marked for retry parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash b392e08abf3b562aafe9e7defecbb6e1 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca544a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 1 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 0 failed parsl.dataflow.dflow INFO: Task 100 marked for retry parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 34627830226198aec8c42ab23a2d89dd parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 1 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 366b16a6f534092e9c0ca74a00ed1914 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 3068f8d35c5760ed81b9f57037fc3e57 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688721a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a0c07868-8341-4c04-b2b2-94da15452a2f_characterizeImage_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a0c07868-8341-4c04-b2b2-94da15452a2f_characterizeImage_1123958_34.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8ed28c84-1a05-40ec-8c15-0861ad02e06c_isr_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8ed28c84-1a05-40ec-8c15-0861ad02e06c_isr_1123958_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 49 try 0 failed parsl.dataflow.dflow INFO: Task 49 marked for retry parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 7149744b5401b989c635a544fcc93d85 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca54ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 1 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 0 failed parsl.dataflow.dflow INFO: Task 64 marked for retry parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 915797eb43c9c476109dcb9fe6070674 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 1 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 133 try 0 failed parsl.dataflow.dflow INFO: Task 133 marked for retry parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 5c21cec00f841413205163a026078beb parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 1 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 73 try 0 failed parsl.dataflow.dflow INFO: Task 73 marked for retry parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash c61a15b407efe1a18fa6bdaf28706531 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca567a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 1 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 0 failed parsl.dataflow.dflow INFO: Task 118 marked for retry parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 03c5fa55de0adea66ef513e35166dc4c parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 1 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 0 failed parsl.dataflow.dflow INFO: Task 124 marked for retry parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 8eab558966ecefd80044f943556fe898 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 1 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 25 try 0 failed parsl.dataflow.dflow INFO: Task 25 marked for retry parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 81f87314cd2d2eea05ca4fe6ba13d991 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 1 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 0 failed parsl.dataflow.dflow INFO: Task 76 marked for retry parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 5ec8a93091cc7028539489f31e6eae2a parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 1 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 205 try 0 failed parsl.dataflow.dflow INFO: Task 205 marked for retry parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 202f87230bbf2c309e5edc9856b456e2 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 1 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef8028e74bc49a63321d7791f0ba815e with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash efa1be6b475a8acc5557333513890141 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688721a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/777917ee-ad9a-43b3-8709-36faa9375245_isr_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/777917ee-ad9a-43b3-8709-36faa9375245_isr_1124087_53.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 55 try 0 failed parsl.dataflow.dflow INFO: Task 55 marked for retry parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 553470b5f4d9ae60e6df7935880cea98 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca554e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 1 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 415 try 0 failed parsl.dataflow.dflow INFO: Task 415 marked for retry parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 5077b83a1c2ad93fb9e16caedfcd81e3 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 1 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 0 failed parsl.dataflow.dflow INFO: Task 1 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash c000c493180d38cfca78582ff6fe4f13 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 1 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 418 try 0 failed parsl.dataflow.dflow INFO: Task 418 marked for retry parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash db114faafb5b568b7d87f36f26b1c706 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1284a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 1 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 334 try 0 failed parsl.dataflow.dflow INFO: Task 334 marked for retry parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 3ff3465232b98d1e68709757d353b81a parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 1 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 331 try 0 failed parsl.dataflow.dflow INFO: Task 331 marked for retry parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 9e58bffff67bb791c13924d14b4ae5d5 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 1 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 412 try 0 failed parsl.dataflow.dflow INFO: Task 412 marked for retry parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash bae99d87cdf984b44621a63899e76f3e parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 1 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 319 try 0 failed parsl.dataflow.dflow INFO: Task 319 marked for retry parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 487f3bb7d6d1e9c23d93b3583c090732 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 1 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 265 try 0 failed parsl.dataflow.dflow INFO: Task 265 marked for retry parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 358b9ee01929f6a95a8ea19df9a5f947 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 1 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stderr 2024-03-07 03:39:04 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:39:04 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:04 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:05 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:05 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:06 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53a788a9c59c82159991ea537fb2d9a with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 5002294924373407b894ccceeafa4111 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9467e45b-b759-49ad-99f0-e78b531eabca_characterizeImage_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9467e45b-b759-49ad-99f0-e78b531eabca_characterizeImage_1123954_31.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b8b23447-1655-4c0a-ac8b-480c24d5e186_isr_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/b8b23447-1655-4c0a-ac8b-480c24d5e186_isr_1123954_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:06 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:06 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:07 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9045a10749e913f35c2b59cdb0c7d24 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 6834881ba9fc070da018d95407d6c1d9 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0f8b126e-303f-4282-8c81-0821b9e805dc_isr_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/0f8b126e-303f-4282-8c81-0821b9e805dc_isr_1124087_42.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a026c4e194eb035c9089f21d07b4fe with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 25b8ceb4f5019a641878f19f8bbaeb59 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:07 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/4d882cf5-8ee0-47e8-9071-f285a95db116_characterizeImage_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/4d882cf5-8ee0-47e8-9071-f285a95db116_characterizeImage_1123958_25.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c14cf2a7-d8d2-4bb7-8193-f914f5b80c25_isr_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/c14cf2a7-d8d2-4bb7-8193-f914f5b80c25_isr_1123958_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:07 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:08 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 170d457ab7daff7eb94f19a56ca77908 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash df670a7a974ff073f3df864693f0a822 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3bfa88a7-8dac-44c1-830e-9634a126d696_characterizeImage_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3bfa88a7-8dac-44c1-830e-9634a126d696_characterizeImage_1123958_6.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8c21272b-969e-4903-8ac7-51b3d25d59d6_isr_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/8c21272b-969e-4903-8ac7-51b3d25d59d6_isr_1123958_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:08 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:09 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:09 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 292 try 0 failed parsl.dataflow.dflow INFO: Task 292 marked for retry parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 9b11d8c391f1add22d2e21dd6bc55f26 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 1 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:09 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 301 try 0 failed parsl.dataflow.dflow INFO: Task 301 marked for retry parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash dd4a8f92fa928307b9e41bd81bd2bdfe parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 1 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:10 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:10 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:10 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:11 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:11 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:11 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:12 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:12 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:12 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:13 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:13 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:13 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:14 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:14 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:14 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:15 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 445 try 0 failed parsl.dataflow.dflow INFO: Task 445 marked for retry parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash f503c85b76021efdc5f2bcc713bdc08d parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 1 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:15 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:16 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:16 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:16 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:17 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:17 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:17 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:18 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:18 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:18 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:19 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:19 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:20 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:20 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:20 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 241 try 0 failed parsl.dataflow.dflow INFO: Task 241 marked for retry parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 077a92be8651eb7013542b5d4fdebe42 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 1 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:21 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59d3e71db3b62e9db5490ef362ff27d3 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 23105cf2a04303904f0f1a23d72906ad parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/48e969f6-0671-4108-864b-24691d29e6e5_characterizeImage_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/48e969f6-0671-4108-864b-24691d29e6e5_characterizeImage_1123958_21.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/309472f6-0dfa-407c-8db5-bd927a1587b5_isr_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123958/309472f6-0dfa-407c-8db5-bd927a1587b5_isr_1123958_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:21 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:22 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13fccdffc99363e0ebc6c78553ab0131 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 5b350e802da89d3a6ed368f12907824b parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3e1e743-f182-45aa-b0a3-626a12936e18_isr_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/f3e1e743-f182-45aa-b0a3-626a12936e18_isr_1124087_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:22 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:22 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15ffc855f0c5bdf6784226655149b4f3 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 8c8207ae6ecd63d1d51e1bd962df9a8d parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f3660c5c-b3c7-41ab-a673-60bf516a53c1_characterizeImage_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f3660c5c-b3c7-41ab-a673-60bf516a53c1_characterizeImage_1123954_56.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f4f1569-2407-4f4c-960f-9c138b9d8139_isr_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/6f4f1569-2407-4f4c-960f-9c138b9d8139_isr_1123954_56.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f10e575bb39d51d2c0228e453ca0d3e with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 4e15668ced3c74de525b2db8c9c6d4e2 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ca075393-1115-43c0-8b71-5115b4309c19_characterizeImage_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ca075393-1115-43c0-8b71-5115b4309c19_characterizeImage_1123956_31.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/fff04bc1-131c-499d-9d66-faebdfe0066f_isr_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123956/fff04bc1-131c-499d-9d66-faebdfe0066f_isr_1123956_31.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7baf958b49b5fce7dfafff386a071cb9 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 5cbe71f934415227c4a254535305c8ce parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/23b33197-04ae-4b6d-9c8b-1522048c36a2_characterizeImage_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/23b33197-04ae-4b6d-9c8b-1522048c36a2_characterizeImage_1123954_11.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/db76f1b9-9cae-468c-a16f-be179ab96712_isr_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1123954/db76f1b9-9cae-468c-a16f-be179ab96712_isr_1123954_11.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2f58ed078c57c2d295637aa4adb18c8 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 971fa46d26c1ddde2f6c5eab813ba3e4 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/608be376-8e61-4b62-9f1e-292a4444287b_isr_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/608be376-8e61-4b62-9f1e-292a4444287b_isr_1124087_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:23 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:23 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97939bbcfc3da45ef0d0af776c32d0f5 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 6285ea3f51196c4bee54f7e2ecf04112 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8aea0967-8c1e-41e0-bb1e-696fd37cdc73_isr_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/8aea0967-8c1e-41e0-bb1e-696fd37cdc73_isr_1124087_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:23 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:24 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:24 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76de56a32d90506c0182e4ef5bae25d4 with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash a7bb50e5d8074ea951d1cf4dc9701aa5 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/e6ef47fa-6786-4edb-aaa5-0acbf5e95499_isr_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/isr/1124087/e6ef47fa-6786-4edb-aaa5-0acbf5e95499_isr_1124087_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:24 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:25 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:25 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:26 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:26 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 247 try 0 failed parsl.dataflow.dflow INFO: Task 247 marked for retry parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 7bbf191a2c92df9540b1e43f7a147f5a parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 1 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 2024-03-07 03:39:36 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:39:36 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:37 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:37 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:38 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:38 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:38 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:39 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:39 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:39 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:40 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:40 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:40 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:41 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:41 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 232 try 0 failed parsl.dataflow.dflow INFO: Task 232 marked for retry parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 7e28b64007360f68cbc861ee13394355 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 1 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:41 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:42 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:42 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:43 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:43 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:43 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:44 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:44 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:44 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:45 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:45 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:46 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:46 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:46 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:47 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:47 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:47 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:48 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ca2cb24058a080aa282df8f5fdb6571 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 75a6a849d1a9a01ace0d02bf35976a62 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1ebf18f8-9e7c-45f5-86a9-408224088a5a_calibrate_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1ebf18f8-9e7c-45f5-86a9-408224088a5a_calibrate_1123956_57.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/03d44e55-3e08-4779-897b-51b9dfbf42b3_characterizeImage_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/03d44e55-3e08-4779-897b-51b9dfbf42b3_characterizeImage_1123956_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:48 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:48 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:49 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:49 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:50 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:50 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:50 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:51 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:51 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:51 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:52 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:52 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:52 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d88c07790d02664126770e52548bb35b with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 8f0c908f9b89c7e4d1e9d4de55a3fa74 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9a064421-8484-419c-8b7b-e831b34f8aed_calibrate_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9a064421-8484-419c-8b7b-e831b34f8aed_calibrate_1123958_5.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cebac510-a288-4826-bc34-6a922b459833_characterizeImage_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cebac510-a288-4826-bc34-6a922b459833_characterizeImage_1123958_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:53 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:53 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:54 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:54 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:54 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:55 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:55 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:55 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:56 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:56 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:57 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:57 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:57 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6495d905ba6d150395f6dd6be9a0ca54 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 77364d45ebafc6d3d4b742ea57ea66bc parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/0aecadd2-fc62-4d7a-9adc-fae213fe9da7_calibrate_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/0aecadd2-fc62-4d7a-9adc-fae213fe9da7_calibrate_1123956_38.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2283f775-7f0b-41da-9bd0-0a7bb0e14ed9_characterizeImage_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2283f775-7f0b-41da-9bd0-0a7bb0e14ed9_characterizeImage_1123956_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:58 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4cbb7ee26d3cfb1cd28fd1145a9f6c7 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 61bd79799455d3c1cab6fe4a54461617 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/705897f1-7735-48bd-a5f2-628649a7f2b9_calibrate_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/705897f1-7735-48bd-a5f2-628649a7f2b9_calibrate_1123958_57.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8900e632-f597-4d29-96f8-5ebbb81dbd96_characterizeImage_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8900e632-f597-4d29-96f8-5ebbb81dbd96_characterizeImage_1123958_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:39:58 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f3c9521f8a73aaab6b85a256d2c4da7 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 853b807e8f7d199ea3149ca1f3466932 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc61d3b-008a-40a5-bed9-67f2179c72da_calibrate_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc61d3b-008a-40a5-bed9-67f2179c72da_calibrate_1123958_23.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/72744fcc-a947-4845-950e-51d0961c844d_characterizeImage_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/72744fcc-a947-4845-950e-51d0961c844d_characterizeImage_1123958_23.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95fc0757c2907b22b7a9364616c10b8 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 3ccc7b5522e1b51791c894da8e1da8b3 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/949fce41-e6c5-4950-8c2c-b8623073126e_calibrate_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/949fce41-e6c5-4950-8c2c-b8623073126e_calibrate_1123956_40.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/76a75126-88d9-4a0a-b861-d87131eddb56_characterizeImage_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/76a75126-88d9-4a0a-b861-d87131eddb56_characterizeImage_1123956_40.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 206980ffa999d42b0a2af0956055c46c with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 91ad19b921b3396663b2b24ea71d25d6 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/65c9d02e-9918-49ca-8d86-1fce8a1ae631_characterizeImage_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/65c9d02e-9918-49ca-8d86-1fce8a1ae631_characterizeImage_1123958_31.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f852b6b7743d2acb47ad78e34abec803 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 33f70c59387f5611cdd73c937fd33c5b parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9801a45f-d654-4b71-a1d0-333e20cd08f2_calibrate_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9801a45f-d654-4b71-a1d0-333e20cd08f2_calibrate_1123956_45.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/81ea7ca2-6583-4d38-865a-01ba50492949_characterizeImage_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/81ea7ca2-6583-4d38-865a-01ba50492949_characterizeImage_1123956_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:00 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a95c3e7506ea53d1302d0fa7cba34f79 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash fb06a8c4a15c6e489e8c68b32bb15f6c parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca562a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1b1235bd-c631-4604-a43a-11337a65efaa_calibrate_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1b1235bd-c631-4604-a43a-11337a65efaa_calibrate_1123956_30.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae50c8de-094a-42de-9567-f76b087f2ecc_characterizeImage_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae50c8de-094a-42de-9567-f76b087f2ecc_characterizeImage_1123956_30.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96de4f8b253dd9a2bed047399aad20bb with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 70e57752a0f37b0d9f66b02ac2a3d75d parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/83b3b14f-a546-48c9-936e-aee574d8bd1f_calibrate_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/83b3b14f-a546-48c9-936e-aee574d8bd1f_calibrate_1123956_17.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b271d02d-e20f-48c7-9a7d-9de8ff0de76f_characterizeImage_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b271d02d-e20f-48c7-9a7d-9de8ff0de76f_characterizeImage_1123956_17.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea0df0b9aa43264addbc5d651ed98961 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash e51542146c482f2ef1a3d023ac199e5b parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/815181a2-9292-4bbe-8b60-f754670d53b5_calibrate_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/815181a2-9292-4bbe-8b60-f754670d53b5_calibrate_1123954_50.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/19872edb-fb88-4fec-bb9c-779d8c7ae462_characterizeImage_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/19872edb-fb88-4fec-bb9c-779d8c7ae462_characterizeImage_1123954_50.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff146db96e85f01d98079834218ea0a3 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash f4daa399be1c1b19f2febb1e92e67af3 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/366b0ea8-24be-436c-97fc-d37a7ed5ce2e_calibrate_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/366b0ea8-24be-436c-97fc-d37a7ed5ce2e_calibrate_1123956_7.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83184d14-5f6a-452c-b57a-d046a6f331aa_characterizeImage_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83184d14-5f6a-452c-b57a-d046a6f331aa_characterizeImage_1123956_7.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56b38d1d95ca4b0219ac1465852333b7 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 511b6d8030886d24418b85fdbadb73cd parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/681d9e66-37fc-41e0-9ef7-c32d06220f5a_calibrate_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/681d9e66-37fc-41e0-9ef7-c32d06220f5a_calibrate_1123956_4.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/27399d5f-be61-4071-a646-a16317e6e2cd_characterizeImage_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/27399d5f-be61-4071-a646-a16317e6e2cd_characterizeImage_1123956_4.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3b07580d0c5fb9745a5e2890346d7f4 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 2b992e1b3b99ecccf3516d07e09eb645 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/11064b60-d210-4222-92bc-a35d009e23f8_calibrate_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/11064b60-d210-4222-92bc-a35d009e23f8_calibrate_1123958_49.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/bebccd5e-ecb1-46f1-9340-4c399de71118_characterizeImage_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/bebccd5e-ecb1-46f1-9340-4c399de71118_characterizeImage_1123958_49.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d38f260afe21ef5f8f1360b491222803 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 6ff7ac33814f13ed4072e0169e215eb0 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5703dc4-03da-480d-b13b-26d6932fe836_calibrate_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5703dc4-03da-480d-b13b-26d6932fe836_calibrate_1123954_40.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c50c0a67-3244-4790-a9f5-0f474ce03a6e_characterizeImage_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c50c0a67-3244-4790-a9f5-0f474ce03a6e_characterizeImage_1123954_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 466 try 0 failed parsl.dataflow.dflow INFO: Task 466 marked for retry parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 83436b1bb1675282fef2ea3ec779392c parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 1 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 749bfd5514d2addf53bfc74795b330bb with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 742b88f7bcf26c8a6084ac60d5ea8940 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/925bacec-f40a-42e8-a537-b04b020d4b6d_calibrate_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/925bacec-f40a-42e8-a537-b04b020d4b6d_calibrate_1123958_12.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ffe34f4d-6004-48e0-9e22-035d095815c7_characterizeImage_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ffe34f4d-6004-48e0-9e22-035d095815c7_characterizeImage_1123958_12.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66ad6fbc5f428cb1cb5eb9ab165fe04a with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 184713d2b907cfd9978eb0312b39d834 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca542c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d3d38078-588e-47eb-848e-096c4b6b11b4_calibrate_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d3d38078-588e-47eb-848e-096c4b6b11b4_calibrate_1123958_45.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f83735cd-a35d-4b6c-8596-fd11f2022e62_characterizeImage_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f83735cd-a35d-4b6c-8596-fd11f2022e62_characterizeImage_1123958_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 490 try 0 failed parsl.dataflow.dflow INFO: Task 490 marked for retry parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 363b64fdab6918421123bfca0735f94a parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1716c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 1 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20719a6f8a9b30aad5c86ae11b0440ae with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 933398653b14d0c5e2c9aaf564f2af72 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a82d629-603d-44b6-b8ab-b725bd85eb48_calibrate_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a82d629-603d-44b6-b8ab-b725bd85eb48_calibrate_1123956_10.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2d477768-1e22-4c31-9a33-ad5f68d3f0c7_characterizeImage_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/2d477768-1e22-4c31-9a33-ad5f68d3f0c7_characterizeImage_1123956_10.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e0d799b2a45b49a74ea8ffe5627c88 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash ae559617b2d04171503394cf4965618c parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0a7e2fa2-0580-4d69-bfce-f072ee24443e_calibrate_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0a7e2fa2-0580-4d69-bfce-f072ee24443e_calibrate_1123954_41.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/eb102921-10c5-446d-a1bd-1b469d0f4268_characterizeImage_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/eb102921-10c5-446d-a1bd-1b469d0f4268_characterizeImage_1123954_41.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56dd8afe4b248f4a7df2ec6c66780e85 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash ab7b8a5cbc5f7ae44eb443e69ed9e36d parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f34c647f-4d7c-4af8-b28c-6ad0ceb45d65_calibrate_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f34c647f-4d7c-4af8-b28c-6ad0ceb45d65_calibrate_1123954_30.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/4889cdd0-26b7-4f93-a092-69a1685513f6_characterizeImage_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/4889cdd0-26b7-4f93-a092-69a1685513f6_characterizeImage_1123954_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 487 try 0 failed parsl.dataflow.dflow INFO: Task 487 marked for retry parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 47f4d4447ab1302d3ca6abd182923b02 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 1 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13b7ece65f0f9218fd3707b9dcaeddb1 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 68611d73e989f39100db4020050d69e6 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8ed23f33-0bfc-4126-b792-cc72ded12cd7_calibrate_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8ed23f33-0bfc-4126-b792-cc72ded12cd7_calibrate_1123958_3.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/6e09f4ba-182e-440f-81f1-fbd8b6f84976_characterizeImage_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/6e09f4ba-182e-440f-81f1-fbd8b6f84976_characterizeImage_1123958_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 472 try 0 failed parsl.dataflow.dflow INFO: Task 472 marked for retry parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash c66aaadbf21907522ac40b342a1e59a8 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 1 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8eff63a3e794af3ec256c8a9c2c6540 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash b59827f959eb5948690984e1e5e8938c parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2512d812-1576-41a2-975a-b50de224c617_calibrate_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2512d812-1576-41a2-975a-b50de224c617_calibrate_1123958_58.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/47702bda-15dc-4cd8-a515-57c309d9c787_characterizeImage_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/47702bda-15dc-4cd8-a515-57c309d9c787_characterizeImage_1123958_58.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3cf38393958cc041f2f07fa13db966a with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 0f049cc302af2bb2e1f32f74aa777c7e parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/85aeebe7-9570-4398-a155-7931368f6a4d_calibrate_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/85aeebe7-9570-4398-a155-7931368f6a4d_calibrate_1123954_47.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/67efeb9f-a7f9-4e65-a2b6-528e96ed477d_characterizeImage_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/67efeb9f-a7f9-4e65-a2b6-528e96ed477d_characterizeImage_1123954_47.stderr 2024-03-07 03:40:10 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:40:10 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:11 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca278d9909edd00f367e9d212cdb0cdb with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 745018cc608a6b2cf4a5e13d9bb4f365 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b8704ac9-76bd-4585-9e59-fab8b644a334_calibrate_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b8704ac9-76bd-4585-9e59-fab8b644a334_calibrate_1123956_35.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/001d1e48-7220-411d-9761-3d6b6cc744b9_characterizeImage_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/001d1e48-7220-411d-9761-3d6b6cc744b9_characterizeImage_1123956_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:11 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08b714b37b9c788db119d665e096d5d6 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash ce7f561deed000581c63402769d0a8d5 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/021dfc23-99cb-400a-b92d-7e7c8db5129d_calibrate_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/021dfc23-99cb-400a-b92d-7e7c8db5129d_calibrate_1123954_42.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c3554b5e-713f-47ae-92d8-10cb4c06fa3d_characterizeImage_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/c3554b5e-713f-47ae-92d8-10cb4c06fa3d_characterizeImage_1123954_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:12 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:12 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47b51ed80d86ec0e85a1f8a16cedcbc3 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 379feab4d8f2b8b12a258a54a011cecd parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1299e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/252b7a2d-b964-4523-aec9-1b469efbc51c_calibrate_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/252b7a2d-b964-4523-aec9-1b469efbc51c_calibrate_1123958_9.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b44a3808-6e1c-4449-8b28-367404cefc75_characterizeImage_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b44a3808-6e1c-4449-8b28-367404cefc75_characterizeImage_1123958_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:12 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 565abf8e8232ac0badb4aed8580bf732 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 93ad45a02fed0c89e67d58c411dc7355 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4af15a9b-aa94-484d-8156-3de1ad9e33cf_calibrate_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4af15a9b-aa94-484d-8156-3de1ad9e33cf_calibrate_1123958_30.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/68922a5c-4198-4779-93cc-7572bdf7926d_characterizeImage_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/68922a5c-4198-4779-93cc-7572bdf7926d_characterizeImage_1123958_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:13 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f998674c2f1dcb933e571040ad11615 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash d2b5e0a755002e64a75d3676a55a9099 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b705e1ad-7c88-4037-a36e-ff78913a8ca0_calibrate_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b705e1ad-7c88-4037-a36e-ff78913a8ca0_calibrate_1123958_39.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d98bc6c4-eae6-40c2-ad6f-0c69dc0d48c9_characterizeImage_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d98bc6c4-eae6-40c2-ad6f-0c69dc0d48c9_characterizeImage_1123958_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:13 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:13 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:13 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:13 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7805762b40c9a85ff130c9ff1d0580d6 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 819bb9dc9535673193f8c398945b5820 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/920a4d47-0957-49d3-9984-76565e4d1bb4_calibrate_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/920a4d47-0957-49d3-9984-76565e4d1bb4_calibrate_1123954_16.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7bfac662-56dc-4c61-9402-2a4359ad8299_characterizeImage_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7bfac662-56dc-4c61-9402-2a4359ad8299_characterizeImage_1123954_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:14 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:14 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:14 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:14 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:14 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:14 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:14 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aac664482a6ed89b99414f36541e1eff with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash e7b0ac6a77dffd5a2b75ed1ce0cab8b7 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8936eb21-2564-408f-b10a-006657e0bb7e_calibrate_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8936eb21-2564-408f-b10a-006657e0bb7e_calibrate_1123958_56.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/aa0c26fe-e098-46a4-8c3d-4555f11119b7_characterizeImage_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/aa0c26fe-e098-46a4-8c3d-4555f11119b7_characterizeImage_1123958_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:15 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:15 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:15 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b19b8f064e4dfc9c90d63e2eaba1589 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 84d914368bb4a738d2615e2756d3647a parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e9f9278d-0f84-4a99-872c-acf3f465a52b_calibrate_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e9f9278d-0f84-4a99-872c-acf3f465a52b_calibrate_1123954_17.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2a17e544-37ae-45b6-b64a-6f1e6559d00f_characterizeImage_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2a17e544-37ae-45b6-b64a-6f1e6559d00f_characterizeImage_1123954_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:15 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:16 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a1ff1ad54c9a5509ed7d083a77994f5 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 075ecd3dab71eeb1bba1d8e34931eee5 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/cb6bed30-baf7-4328-b420-e5aeecc30358_calibrate_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/cb6bed30-baf7-4328-b420-e5aeecc30358_calibrate_1123954_43.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fc301d8c-b866-4c66-8c8d-08daab99fcde_characterizeImage_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fc301d8c-b866-4c66-8c8d-08daab99fcde_characterizeImage_1123954_43.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 574 try 0 failed parsl.dataflow.dflow INFO: Task 574 marked for retry parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 3fb2648f636aa96f82dd546aee16de2e parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 1 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 529 try 0 failed parsl.dataflow.dflow INFO: Task 529 marked for retry parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 8970d5e191d44c57151af1768dd92906 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 1 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:16 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:16 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:16 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3874c8b99cd5ce0654066bdbd7db990a with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 8f224d101170a1919d232bda50881504 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6eed9cc0-dc76-498b-a4fb-13f09d4f5cf9_calibrate_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6eed9cc0-dc76-498b-a4fb-13f09d4f5cf9_calibrate_1123958_54.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c1b5e331-343c-40ee-a234-fd7caa10750c_characterizeImage_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c1b5e331-343c-40ee-a234-fd7caa10750c_characterizeImage_1123958_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:17 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0c5827479050d8cba88b0d7c8f742b with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 5100260503e5ee9aa42866289d90368d parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/92bcf368-ea65-4c9b-b0b1-14eaa81253c9_calibrate_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/92bcf368-ea65-4c9b-b0b1-14eaa81253c9_calibrate_1123958_37.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e8044f82-a42d-49bb-a979-b27622003b8d_characterizeImage_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e8044f82-a42d-49bb-a979-b27622003b8d_characterizeImage_1123958_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:17 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:17 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:17 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:17 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:17 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:17 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 688 try 0 failed parsl.dataflow.dflow INFO: Task 688 marked for retry parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 5e196acf326a5f34a6381ef10c3f3581 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 1 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:18 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4aa527b06aa66d0b41e4cf77de76d9f with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash a0b672ccd51d236aa154502bb838c039 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c696acbd-d97f-4607-8c28-5ae5e5f5babe_calibrate_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c696acbd-d97f-4607-8c28-5ae5e5f5babe_calibrate_1123954_48.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1acaae9e-4843-4a2c-bebe-1c77a8dd3a87_characterizeImage_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1acaae9e-4843-4a2c-bebe-1c77a8dd3a87_characterizeImage_1123954_48.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aca6d2112cc94aef2bdde2eadbfffdc with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash e73be789cfea528f945251e816546ff4 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/ffccb6f1-e967-4c2e-b759-fbbda93173f6_calibrate_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/ffccb6f1-e967-4c2e-b759-fbbda93173f6_calibrate_1123958_28.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f289007c-b4e4-4272-bc29-6a3df2ab3cd5_characterizeImage_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f289007c-b4e4-4272-bc29-6a3df2ab3cd5_characterizeImage_1123958_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:18 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 589 try 0 failed parsl.dataflow.dflow INFO: Task 589 marked for retry parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash ce6b71be7acce86cecfcca01dde72da0 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 1 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:18 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23815e0fb2384d49933fbbc93723d527 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 0af3d35ed3e45ae837a6d3959e758c2e parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/76f4bfb0-764d-403b-b11f-ddc4ed8f8ac5_calibrate_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/76f4bfb0-764d-403b-b11f-ddc4ed8f8ac5_calibrate_1123958_22.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/5a84869e-a9fb-4cf1-9b36-8abbf1781aaf_characterizeImage_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/5a84869e-a9fb-4cf1-9b36-8abbf1781aaf_characterizeImage_1123958_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:18 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:19 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:19 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:19 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:19 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:19 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 144e86126e7076711615e6728a43e4b9 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 1f56b19465c80674c0e3edb51525d4de parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1de7179-439a-428d-8fbe-21d927cfc40e_calibrate_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1de7179-439a-428d-8fbe-21d927cfc40e_calibrate_1123954_23.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3d2113f8-c13c-4ada-b7eb-aae81c13418b_characterizeImage_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3d2113f8-c13c-4ada-b7eb-aae81c13418b_characterizeImage_1123954_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:19 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:19 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:20 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:20 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeb364e008e6bf25ba4cd7d79cda550e with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 9491652ccf51282d8a88ded5450e2ea1 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e38973ce-250c-4b56-a7ee-fada127fc56e_calibrate_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e38973ce-250c-4b56-a7ee-fada127fc56e_calibrate_1123954_46.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0bbcb0dc-21f8-42e7-9ebb-d10fe8b03feb_characterizeImage_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0bbcb0dc-21f8-42e7-9ebb-d10fe8b03feb_characterizeImage_1123954_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:20 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0522f7977fc5c4597cd2e6a761d55873 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 058fa20089f0a709d59277db0396c23f parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/38816df2-756b-4533-acc2-8f84ddd6850a_calibrate_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/38816df2-756b-4533-acc2-8f84ddd6850a_calibrate_1123956_41.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c7f40355-8c24-4eb6-b3f7-846fef8c1845_characterizeImage_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c7f40355-8c24-4eb6-b3f7-846fef8c1845_characterizeImage_1123956_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:21 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:21 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:21 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:22 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:22 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:22 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 613 try 0 failed parsl.dataflow.dflow INFO: Task 613 marked for retry parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 24b5f9a1817d60ccca15f2d84dd4de95 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 1 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:23 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0b74cba647a4e011d1a83963a3ce6b8 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash b63695bd2181a0b26b1200fba664e0c5 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/47ae67b5-d348-442d-a349-9b4ef8ba95ba_calibrate_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/47ae67b5-d348-442d-a349-9b4ef8ba95ba_calibrate_1123954_49.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fe8f6e1-f771-44c1-a894-41de2d7eb0c8_characterizeImage_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fe8f6e1-f771-44c1-a894-41de2d7eb0c8_characterizeImage_1123954_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:23 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:23 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 532 try 0 failed parsl.dataflow.dflow INFO: Task 532 marked for retry parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 45d6a8ba54f5e6ee2c28eb1095b94851 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1737e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 1 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e104851f4ad85a20b1ba49f8a1407deb with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash c3785408078ddf8da4a41c61d57e0b92 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e8c128fb-48d3-4f38-bf18-70861ad5d539_calibrate_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e8c128fb-48d3-4f38-bf18-70861ad5d539_calibrate_1123956_25.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/13ac5d55-50db-4aba-bdd9-ad1b0fc96349_characterizeImage_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/13ac5d55-50db-4aba-bdd9-ad1b0fc96349_characterizeImage_1123956_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:24 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:24 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:24 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:24 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:24 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 697 try 0 failed parsl.dataflow.dflow INFO: Task 697 marked for retry parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 15e2b6d411ee96194d43bafeb8a46a56 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 1 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e4814f951c8d821f7c5797973a54329 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash d3ee498c5a4bdb965116b986cb6c00e6 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/4c606afb-fb32-4a38-811e-e26fac9500b3_calibrate_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/4c606afb-fb32-4a38-811e-e26fac9500b3_calibrate_1123954_3.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a404a1f8-5cdf-4b43-ac7f-8f06b3906581_characterizeImage_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a404a1f8-5cdf-4b43-ac7f-8f06b3906581_characterizeImage_1123954_3.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fec6ed580c9b9f9517e191b0899804dd with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash a648495086374689b71a01a074acb8d2 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1b24a157-83d9-4db7-a323-46a8c8e332b2_calibrate_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1b24a157-83d9-4db7-a323-46a8c8e332b2_calibrate_1123958_17.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f3e61f51-4378-462c-8329-b8250d5c5d77_characterizeImage_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f3e61f51-4378-462c-8329-b8250d5c5d77_characterizeImage_1123958_17.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dafbac4285bbf0db6addc9dc9e7aab46 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash b73c3a899ba5db2dc0d09cba6e484212 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/94a79782-0355-43c1-b875-5bf6efae70f5_calibrate_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/94a79782-0355-43c1-b875-5bf6efae70f5_calibrate_1123956_32.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b90c526d-b36c-419f-a549-f9a178673659_characterizeImage_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b90c526d-b36c-419f-a549-f9a178673659_characterizeImage_1123956_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 598 try 0 failed parsl.dataflow.dflow INFO: Task 598 marked for retry parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 16fdf632038b58740a74668b6c9d0824 parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 1 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f3472f926f1ebedf9f9f1268bd3a855 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash de585e852ac156090caf58717ebd90ff parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/518ba6cd-2900-44b0-9b85-0bda526de457_calibrate_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/518ba6cd-2900-44b0-9b85-0bda526de457_calibrate_1123958_51.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/116c7e82-b5c3-41e1-90f2-d907191b13b7_characterizeImage_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/116c7e82-b5c3-41e1-90f2-d907191b13b7_characterizeImage_1123958_51.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f7f315e411f79e6dd2ad7cbbf332a0e with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 9359b08b4cbc13e9df4a3fa8835bf588 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/dc9099b5-c418-479e-859e-be93f55a3e02_calibrate_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/dc9099b5-c418-479e-859e-be93f55a3e02_calibrate_1123958_36.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/1cca87b8-5e76-4ed4-95a4-06ef65fab94f_characterizeImage_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/1cca87b8-5e76-4ed4-95a4-06ef65fab94f_characterizeImage_1123958_36.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f894d4b850d9e994e0980e14cc6cb97d with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 03779c58a5a7df31c5d5a341f42f58a2 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f1269283-56ab-4c3e-a5e8-4ea9bfe11140_calibrate_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f1269283-56ab-4c3e-a5e8-4ea9bfe11140_calibrate_1123958_11.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/eed52d84-5882-457e-8402-ab8c5a497f04_characterizeImage_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/eed52d84-5882-457e-8402-ab8c5a497f04_characterizeImage_1123958_11.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae5040ca36b20d7f0c7b23dfd71437e2 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 2e11826d4112b7981d8cac5092e09cc7 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ec6fdc32-f80d-4c5f-a142-815458209655_calibrate_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ec6fdc32-f80d-4c5f-a142-815458209655_calibrate_1123956_26.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6095d215-259e-462b-a873-bfea51af6078_characterizeImage_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6095d215-259e-462b-a873-bfea51af6078_characterizeImage_1123956_26.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1009efc93f27e946726524387e062c17 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 356e4f5e2a1b5487b84ef08e21a20e90 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/090a2508-cf75-4e94-a345-4acd230ba515_calibrate_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/090a2508-cf75-4e94-a345-4acd230ba515_calibrate_1123956_11.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96bdec32-7a93-4582-8071-9b3fb86e96ff_characterizeImage_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96bdec32-7a93-4582-8071-9b3fb86e96ff_characterizeImage_1123956_11.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a29c7f94e66b1b67e11a7eca27e760a2 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 2476573dfc79c900e8be8449e984fec7 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/164e6172-a1ac-4ab8-976a-971b226cc3a7_calibrate_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/164e6172-a1ac-4ab8-976a-971b226cc3a7_calibrate_1123954_29.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/81284995-f663-4c09-9a7b-c5fa00b28416_characterizeImage_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/81284995-f663-4c09-9a7b-c5fa00b28416_characterizeImage_1123954_29.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cba8c46a7457fd0a042afaff70a4077 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 0c93abf94c3e754b8e427cc8db310546 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8747c728-9618-4175-a9b2-bd478be58843_calibrate_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8747c728-9618-4175-a9b2-bd478be58843_calibrate_1123958_15.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/542f0806-fa57-4df8-9745-3398559eadf6_characterizeImage_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/542f0806-fa57-4df8-9745-3398559eadf6_characterizeImage_1123958_15.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64dd7256f42c78630eacc814d4d1082a with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 57c8c479cf660299092fcdcb1183b5c4 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0d22d6ec-7fe3-4210-aede-8c2689b2aecb_calibrate_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0d22d6ec-7fe3-4210-aede-8c2689b2aecb_calibrate_1123954_19.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/b8ce25b6-6f9c-48e9-ae23-c547289ffc3d_characterizeImage_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/b8ce25b6-6f9c-48e9-ae23-c547289ffc3d_characterizeImage_1123954_19.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40b929bd068debc3260d16e1f773b51e with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 82cff02b0ac0d441156a5852d2aa592d parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/13aa2b05-9cb9-49fe-961a-893fea03bdd8_calibrate_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/13aa2b05-9cb9-49fe-961a-893fea03bdd8_calibrate_1123954_36.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7ed490b3-24a1-463d-9c6b-ed9853c536d2_characterizeImage_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7ed490b3-24a1-463d-9c6b-ed9853c536d2_characterizeImage_1123954_36.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 577 try 0 failed parsl.dataflow.dflow INFO: Task 577 marked for retry parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash f0f399c40fcb3910e2bafb7f71b9e285 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 1 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 640 try 0 failed parsl.dataflow.dflow INFO: Task 640 marked for retry parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 0755a13a0b3a9c1581f38e73d6222198 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 1 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 628 try 0 failed parsl.dataflow.dflow INFO: Task 628 marked for retry parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 196d136254572d1065715edbf8821ef2 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 1 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e613144884e20f794e9499d4b1b3260a with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 963c501507c4256937577b725b45a487 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/fc55c613-b0f8-49cc-81ff-55953684f687_calibrate_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/fc55c613-b0f8-49cc-81ff-55953684f687_calibrate_1123954_18.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a3ec7e6b-505b-42ef-8eef-86406f1b677b_characterizeImage_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a3ec7e6b-505b-42ef-8eef-86406f1b677b_characterizeImage_1123954_18.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 625 try 0 failed parsl.dataflow.dflow INFO: Task 625 marked for retry parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash ecfd50c7ba4321186e68d885e4391cd3 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 1 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 248 parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c33f0cd2e427bd189532b143f1a5b2c with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 580168d4b7a60a50e70ba3551982b47a parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/6a4994d0-91b0-47c1-aad6-5fea3818552e_calibrate_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/6a4994d0-91b0-47c1-aad6-5fea3818552e_calibrate_1123956_21.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/e135b397-9812-45b5-bebb-e73e6c1bddaf_characterizeImage_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/e135b397-9812-45b5-bebb-e73e6c1bddaf_characterizeImage_1123956_21.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a093b1c60091bf0c90d0596aef07fc35 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 9757fe30d0d8628a807993669ae281e9 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/7090cd44-f099-4e36-bfa0-2b197a6982b5_calibrate_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/7090cd44-f099-4e36-bfa0-2b197a6982b5_calibrate_1123954_1.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fd29101-c2f0-4080-99e3-969b0dce24fd_characterizeImage_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9fd29101-c2f0-4080-99e3-969b0dce24fd_characterizeImage_1123954_1.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ace6588315cc672918fc1d7c250d1534 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 6b746173c5ba4ec785f45c02937adad3 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bf2de8d3-9459-4c41-b299-784297e6072d_calibrate_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bf2de8d3-9459-4c41-b299-784297e6072d_calibrate_1123956_29.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6cb108ef-498b-4386-9d92-714ca362b2b9_characterizeImage_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6cb108ef-498b-4386-9d92-714ca362b2b9_characterizeImage_1123956_29.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 562 try 0 failed parsl.dataflow.dflow INFO: Task 562 marked for retry parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 6648a3efe0e031fc24544e7786ef65cc parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 1 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 379 try 0 failed parsl.dataflow.dflow INFO: Task 379 marked for retry parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 5f048ec3f471c2e4f50e5dcb84b29045 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 1 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24b1ee619b42044e345b3849df8a3965 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 61d7810220a85052058c787ea99513c6 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2b8867a3-cfa2-4092-950e-c519f7eda92d_calibrate_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2b8867a3-cfa2-4092-950e-c519f7eda92d_calibrate_1123956_14.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8bf5fa77-1681-4e03-b2cd-247c411fd834_characterizeImage_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8bf5fa77-1681-4e03-b2cd-247c411fd834_characterizeImage_1123956_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 682 try 0 failed parsl.dataflow.dflow INFO: Task 682 marked for retry parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash a203cd63add2c8a45d6d51685d1283a9 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 1 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9953cc9969843c73e6ef533712896812 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 7aaefafb41630e2d1de875b4b058998a parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/190331a3-0fd9-4a30-8643-a8f832b0bdf0_calibrate_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/190331a3-0fd9-4a30-8643-a8f832b0bdf0_calibrate_1123958_1.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b72a3a03-1be5-4f4b-83e0-8ea9c57334da_characterizeImage_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b72a3a03-1be5-4f4b-83e0-8ea9c57334da_characterizeImage_1123958_1.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f273f7e2b31f124296b8ab2df5bda16f with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 25487ff0e1e2ba59f1a2b98536a39e1a parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/059f69a4-3175-4420-b2df-6204da0c5355_calibrate_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/059f69a4-3175-4420-b2df-6204da0c5355_calibrate_1123956_46.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7ef3ea54-0f16-4711-8c9d-74d628272c1b_characterizeImage_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7ef3ea54-0f16-4711-8c9d-74d628272c1b_characterizeImage_1123956_46.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2c5e36e40180559ebd0cc3ee3130303 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash f6cbe053aaafaa58736ee00008b39e8c parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0fdadd55-8fc6-4439-bd66-01b4782525ef_calibrate_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0fdadd55-8fc6-4439-bd66-01b4782525ef_calibrate_1123954_7.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6b381df5-69a4-47b7-b459-4b6d1e4b84a3_characterizeImage_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6b381df5-69a4-47b7-b459-4b6d1e4b84a3_characterizeImage_1123954_7.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23853043d7d639e9db29e25252c5510f with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 7a0ec5abcbdfc7dc9f08175b8ebf289c parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1293a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/21958c61-1506-41f0-bbf1-cff275140151_calibrate_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/21958c61-1506-41f0-bbf1-cff275140151_calibrate_1123956_19.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/94a26a03-7fa5-434e-911b-00b7d7b892b4_characterizeImage_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/94a26a03-7fa5-434e-911b-00b7d7b892b4_characterizeImage_1123956_19.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72f2be2238c5d04d2aae9e9b935e067b with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash b5539020be09867ea80d52e2e8aac8dd parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4b618195-c53b-4236-aa82-93873f6edeb9_calibrate_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4b618195-c53b-4236-aa82-93873f6edeb9_calibrate_1123958_50.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f35c33a5-77d7-4da1-b9d3-2f1e33aa4d2a_characterizeImage_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f35c33a5-77d7-4da1-b9d3-2f1e33aa4d2a_characterizeImage_1123958_50.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e30bf3af893191d555b0063c97e2f59d with result from task 439 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 246 parsl.dataflow.strategy DEBUG: Executor multi has 246 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash ac088d4ab9a7dc529fe30dfbc1d005c3 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c128f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e2c58479-97b5-4194-b88c-708b4c277f28_calibrate_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e2c58479-97b5-4194-b88c-708b4c277f28_calibrate_1123954_21.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fb4c9005-ff31-4e3f-86a6-63827d39ffa5_characterizeImage_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fb4c9005-ff31-4e3f-86a6-63827d39ffa5_characterizeImage_1123954_21.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c193793a22eede83d836c6ec76b07692 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 1d16416eb486ec9a978a47cbae6c3d92 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/99971dc1-2d01-4f76-a718-a35c8c28dd15_calibrate_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/99971dc1-2d01-4f76-a718-a35c8c28dd15_calibrate_1123954_26.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a7ef0f07-780f-4fb3-ad14-d16c7f0f2e40_characterizeImage_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a7ef0f07-780f-4fb3-ad14-d16c7f0f2e40_characterizeImage_1123954_26.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64e3b4fa6cb74e2874fa047dc81720fb with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 938bd585f197d7529c165427d00c58fe parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d8d86925-a4ce-4cf0-a254-a02f13095036_calibrate_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d8d86925-a4ce-4cf0-a254-a02f13095036_calibrate_1123956_12.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b542ddc5-58d4-4abc-bc99-29ba2ae7ae60_characterizeImage_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b542ddc5-58d4-4abc-bc99-29ba2ae7ae60_characterizeImage_1123956_12.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad80eee1f4ce4d4cac7dabd676463a41 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 33d752cc18567d5e3340e32b503671cd parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873260c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/674c9a65-b0f5-4d10-accf-626a6190a9c2_calibrate_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/674c9a65-b0f5-4d10-accf-626a6190a9c2_calibrate_1123954_58.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/73123d8b-e271-4019-9827-b2eb03c02aa5_characterizeImage_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/73123d8b-e271-4019-9827-b2eb03c02aa5_characterizeImage_1123954_58.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b2f2682a2d585d3d8ec67f6475f8d3c with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 37ed6f6ee0a7006ae24f81cde008de55 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e7c0f3ca-e123-4c32-a05d-214dee52a7eb_calibrate_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e7c0f3ca-e123-4c32-a05d-214dee52a7eb_calibrate_1123956_52.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/877ecc48-0cf1-42e6-b692-177d52d4fb34_characterizeImage_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/877ecc48-0cf1-42e6-b692-177d52d4fb34_characterizeImage_1123956_52.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fb0b9570e1a5e95aaf1ba6f37fe639d with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash f2463773518f9af1b2d9c73fcc0f3179 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b870d33a-390a-4843-a887-4ee657867948_calibrate_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b870d33a-390a-4843-a887-4ee657867948_calibrate_1123954_51.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e028b4a1-7f80-4366-b05b-ed2f3829198a_characterizeImage_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e028b4a1-7f80-4366-b05b-ed2f3829198a_characterizeImage_1123954_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 541 try 0 failed parsl.dataflow.dflow INFO: Task 541 marked for retry parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash a7007ec0a6812f3aab3655f69b526cb6 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873254e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 1 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8609cdf91392899dc6017dff86ec9c4d with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash cbb491882ba33593e52ca13c5924aa56 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5022d58-4829-4c04-ae86-98d8d391e4b9_calibrate_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5022d58-4829-4c04-ae86-98d8d391e4b9_calibrate_1123954_13.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2c6e1906-4e3b-432c-adb0-27b949704996_characterizeImage_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2c6e1906-4e3b-432c-adb0-27b949704996_characterizeImage_1123954_13.stderr 2024-03-07 03:40:45 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:40:45 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:45 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 388 try 0 failed parsl.dataflow.dflow INFO: Task 388 marked for retry parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash f114c20ac0b64ae0ef146a01e003f477 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 1 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41b476119f954a316f3d659c153fdd3a with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 7d6b2ad12e0ee5d08607a02eb9ed6d4b parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca540e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/00356e81-5d7b-4d26-bda4-40cb6862533e_calibrate_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/00356e81-5d7b-4d26-bda4-40cb6862533e_calibrate_1123956_27.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7117b39f-998d-48df-89d6-78e7c5a8dcfb_characterizeImage_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7117b39f-998d-48df-89d6-78e7c5a8dcfb_characterizeImage_1123956_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:45 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:46 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 316b40e60a7be8bddd087f21d116248a with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash b151e053cd03994ff9b25d928a4a50fa parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9f68b607-8728-40e0-9b47-b580076af5b7_calibrate_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9f68b607-8728-40e0-9b47-b580076af5b7_calibrate_1123958_4.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7b65691a-e011-4f03-a945-f552713e3fc4_characterizeImage_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7b65691a-e011-4f03-a945-f552713e3fc4_characterizeImage_1123958_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:46 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a393f85537464a1abed2e760375528a5 with result from task 7 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:46 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 50f1584425a05ba6ecdeff8fb8b0e7a5 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca3d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/7156f0d5-7366-459e-a1dc-b198fb475464_calibrate_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/7156f0d5-7366-459e-a1dc-b198fb475464_calibrate_1123956_3.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0e8dbd00-7db5-497e-a65a-46e967057f96_characterizeImage_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0e8dbd00-7db5-497e-a65a-46e967057f96_characterizeImage_1123956_3.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b74fd8f75b5431659309b36ca78b18f2 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash ce1042bd3899d55d72c5234cbddd742f parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bba558cd-055e-4ea2-b9ef-1693ea20ef63_calibrate_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bba558cd-055e-4ea2-b9ef-1693ea20ef63_calibrate_1123958_44.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f120bacc-f06e-4222-ae7d-53219df6161d_characterizeImage_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f120bacc-f06e-4222-ae7d-53219df6161d_characterizeImage_1123958_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:47 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:47 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:47 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0552d2669e6df8dc2df686791c4377e4 with result from task 169 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:48 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 6b4f42b155308b88c47cf9e22d1ff170 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f29c82e1-c6ba-4f5b-814d-94f8921f09fd_calibrate_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f29c82e1-c6ba-4f5b-814d-94f8921f09fd_calibrate_1123958_10.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cf15b91b-1e78-46c6-ad7d-c025ceeea493_characterizeImage_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cf15b91b-1e78-46c6-ad7d-c025ceeea493_characterizeImage_1123958_10.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3c3f1db9005c5ce5afe7440b1968f5f with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash c147eee541078792d536ac5abb377d69 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6011b5e3-47d1-4fde-ad5f-3988865eb172_calibrate_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6011b5e3-47d1-4fde-ad5f-3988865eb172_calibrate_1123958_42.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b771d235-4ee3-4721-b9c2-8a4b8f9221f4_characterizeImage_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b771d235-4ee3-4721-b9c2-8a4b8f9221f4_characterizeImage_1123958_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:48 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 733 try 0 failed parsl.dataflow.dflow INFO: Task 733 marked for retry parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash a6b612697bef76f74dadc87a271f93a4 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 1 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:48 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 655 try 0 failed parsl.dataflow.dflow INFO: Task 655 marked for retry parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 719af86669e2b5ad28901fca11c249ec parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 1 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:49 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:49 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:50 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:50 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:50 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:51 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:51 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:52 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:52 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:52 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:52 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:53 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:53 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:53 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 1 failed parsl.dataflow.dflow ERROR: Task 34 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e89bad3fbb7a2001293003b481e85847 with result from task 34 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 35 try 0 failed parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/07cac022-9c80-4959-b9ce-267c28692eaa_calibrate_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/07cac022-9c80-4959-b9ce-267c28692eaa_calibrate_1124087_37.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/3f7c1429-d579-41f1-ac8e-3d650aac200d_characterizeImage_1124087_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:54 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 19 try 1 failed parsl.dataflow.dflow ERROR: Task 19 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 79660152977a0c14df444f7ebb7cad75 with result from task 19 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 20 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 20 try 0 failed parsl.dataflow.dflow INFO: Task 20 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:54 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/382696f7-8d20-45c2-8018-3168df5127d4_calibrate_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/382696f7-8d20-45c2-8018-3168df5127d4_calibrate_1124087_54.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/672042f8-c9fe-427c-9514-eb6e2c65e87a_characterizeImage_1124087_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:55 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 1 failed parsl.dataflow.dflow ERROR: Task 52 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ad6166e25c076303099816d1d40bb95c with result from task 52 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 53 try 0 failed parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/4957c1eb-4ac7-493c-a017-e65bc669b30f_calibrate_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/4957c1eb-4ac7-493c-a017-e65bc669b30f_calibrate_1124087_23.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b859e52e-dbb5-4220-bbc3-2651e44fe923_characterizeImage_1124087_23.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 340 try 1 failed parsl.dataflow.dflow ERROR: Task 340 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a515cdccae2b9ce0ea2a92a449e4b3cc with result from task 340 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 341 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 341 try 0 failed parsl.dataflow.dflow INFO: Task 341 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/392e4864-5f9f-44e0-8136-31bfe5d49fde_calibrate_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/392e4864-5f9f-44e0-8136-31bfe5d49fde_calibrate_1124087_27.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/71615165-7f83-4dbd-b40e-6513e60d4bc2_characterizeImage_1124087_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:55 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:55 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 1 failed parsl.dataflow.dflow ERROR: Task 28 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 07e63588df970413f34c38152add60dd with result from task 28 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 29 try 0 failed parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/cb66b271-1461-4616-a445-4186981d2386_calibrate_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/cb66b271-1461-4616-a445-4186981d2386_calibrate_1123956_2.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/3ab8b6a7-0e42-44fe-89d3-9846ed96a506_characterizeImage_1123956_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:56 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:56 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:56 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 217 try 1 failed parsl.dataflow.dflow ERROR: Task 217 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cf93be5b5f6be3415f03e3ce320e0f96 with result from task 217 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 218 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 218 try 0 failed parsl.dataflow.dflow INFO: Task 218 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/1bbafd28-2ab5-4eeb-9c67-54159b6f51ae_calibrate_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/1bbafd28-2ab5-4eeb-9c67-54159b6f51ae_calibrate_1124087_4.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdd98f78-e503-4dcd-8675-e32a13f9c443_characterizeImage_1124087_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:57 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:57 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:58 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 223 try 1 failed parsl.dataflow.dflow ERROR: Task 223 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4cda3dff1034ed6eb3cdd5f0bf7e8a47 with result from task 223 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 224 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 224 try 0 failed parsl.dataflow.dflow INFO: Task 224 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/5dc66d9d-237c-4b8d-b963-e9ad5c4b3d7f_calibrate_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/5dc66d9d-237c-4b8d-b963-e9ad5c4b3d7f_calibrate_1124087_39.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8e71c890-8547-4ebe-9fe3-c22f9edc57c7_characterizeImage_1124087_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:58 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3d9243c5562688e5dd92f6a16aa29b with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 6496ad6a401ebc93523fe8aa11c4b711 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9f99a4be-be7b-41b4-8b58-0e601e972f18_calibrate_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9f99a4be-be7b-41b4-8b58-0e601e972f18_calibrate_1123956_58.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6f54847f-dc90-4e1b-919f-2c1991484186_characterizeImage_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6f54847f-dc90-4e1b-919f-2c1991484186_characterizeImage_1123956_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:58 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:59 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:59 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9f2ad97e42c07e5b760c2352a1f2433 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash c5fa69650e8178ed488c48cf98ae8728 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9d289693-d253-44bb-956c-5dfa24d61553_calibrate_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9d289693-d253-44bb-956c-5dfa24d61553_calibrate_1123958_14.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/384071a2-801a-44c6-a325-89037496a9bf_characterizeImage_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/384071a2-801a-44c6-a325-89037496a9bf_characterizeImage_1123958_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:40:59 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47c0f12e0d3f4a51965309ec4c4b82af with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash a5514599037d040068e8eddd0a2ccba4 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/343aa867-b2e1-4c04-9ce5-ec6e9d80da9a_calibrate_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/343aa867-b2e1-4c04-9ce5-ec6e9d80da9a_calibrate_1123958_26.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f71607de-d951-411c-8fd7-aa8d74466ce6_characterizeImage_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/f71607de-d951-411c-8fd7-aa8d74466ce6_characterizeImage_1123958_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:00 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:00 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:00 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:00 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 220 try 1 failed parsl.dataflow.dflow ERROR: Task 220 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bd845f52772843c76417c5a384d2ccc0 with result from task 220 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 221 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 221 try 0 failed parsl.dataflow.dflow INFO: Task 221 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/9ca941a1-ad39-4c91-b678-3d431a402252_calibrate_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/9ca941a1-ad39-4c91-b678-3d431a402252_calibrate_1124087_8.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/69357de2-3d15-424e-a738-4ddbf8fa3a66_characterizeImage_1124087_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:01 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 226 try 1 failed parsl.dataflow.dflow ERROR: Task 226 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a52f7f454350b07705c818f2aadbd17d with result from task 226 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 227 try 0 failed parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ae5a7f80-e3c1-42ec-a7a3-ebf667628797_calibrate_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ae5a7f80-e3c1-42ec-a7a3-ebf667628797_calibrate_1124087_25.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8b6b4897-7dea-454a-8043-02e9504d696f_characterizeImage_1124087_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:01 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 256 try 1 failed parsl.dataflow.dflow ERROR: Task 256 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d556701645b0eb4bcbfa87cdf609571c with result from task 256 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 257 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 257 try 0 failed parsl.dataflow.dflow INFO: Task 257 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b4b744bc-3519-4b65-b38f-69878f96049d_calibrate_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b4b744bc-3519-4b65-b38f-69878f96049d_calibrate_1124087_36.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/bf2feae3-62c9-4803-9515-2e93662fe124_characterizeImage_1124087_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:01 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:02 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:02 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 1 failed parsl.dataflow.dflow ERROR: Task 100 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 34627830226198aec8c42ab23a2d89dd with result from task 100 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 101 try 0 failed parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/66813df9-23db-4b19-ac8b-e1ad8669e842_calibrate_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/66813df9-23db-4b19-ac8b-e1ad8669e842_calibrate_1124087_11.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c2c56e62-8e41-47de-b3ef-ca49c5550340_characterizeImage_1124087_11.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3bfc1dc081e278c9d433b830dc085a7 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash e125720385eb4c67516d61beccf14ce7 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8a748d1a-a7aa-4bd4-86de-dcabea4163cb_calibrate_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8a748d1a-a7aa-4bd4-86de-dcabea4163cb_calibrate_1123958_27.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/97739230-d1a1-4f2c-96f7-d3b430a19053_characterizeImage_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/97739230-d1a1-4f2c-96f7-d3b430a19053_characterizeImage_1123958_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:04 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:05 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 1 failed parsl.dataflow.dflow ERROR: Task 40 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b392e08abf3b562aafe9e7defecbb6e1 with result from task 40 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 41 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 41 try 0 failed parsl.dataflow.dflow INFO: Task 41 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/511fa3bb-932d-41fd-9650-c719b0c60672_calibrate_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/511fa3bb-932d-41fd-9650-c719b0c60672_calibrate_1124087_43.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0b18baf3-c520-456b-9fb5-21083e6669b4_characterizeImage_1124087_43.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 1 failed parsl.dataflow.dflow ERROR: Task 64 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 915797eb43c9c476109dcb9fe6070674 with result from task 64 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 65 try 0 failed parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/eebfa6dd-7f36-4e4f-9405-6bef2309f277_calibrate_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/eebfa6dd-7f36-4e4f-9405-6bef2309f277_calibrate_1124087_5.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a6720dc8-2c62-4bcd-aca8-8e7099cfab26_characterizeImage_1124087_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:05 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fbd4600444e0334f01ea764e1119cce with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 529bf28e55e0b4ef4679a7804a13d08b parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0686fa32-3e9f-448a-bbfe-ca5053017ee3_calibrate_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0686fa32-3e9f-448a-bbfe-ca5053017ee3_calibrate_1123954_9.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7a7880bc-ab1c-4a63-9027-bf3c02199de8_characterizeImage_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7a7880bc-ab1c-4a63-9027-bf3c02199de8_characterizeImage_1123954_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:05 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 49 try 1 failed parsl.dataflow.dflow ERROR: Task 49 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7149744b5401b989c635a544fcc93d85 with result from task 49 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 50 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 50 try 0 failed parsl.dataflow.dflow INFO: Task 50 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/6553bfdb-5fe5-4f2b-94e1-b3f6184ded54_calibrate_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/6553bfdb-5fe5-4f2b-94e1-b3f6184ded54_calibrate_1124087_20.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/308d24e5-e591-40ac-8db2-b92681bb2ec7_characterizeImage_1124087_20.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 133 try 1 failed parsl.dataflow.dflow ERROR: Task 133 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5c21cec00f841413205163a026078beb with result from task 133 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 134 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 134 try 0 failed parsl.dataflow.dflow INFO: Task 134 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/a683d38a-f0f3-4415-b1b2-b7764b383cca_calibrate_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/a683d38a-f0f3-4415-b1b2-b7764b383cca_calibrate_1124087_2.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9c40b506-9eea-4c23-b157-e49ae264465d_characterizeImage_1124087_2.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 236315fc6aa79ee407b7dd3f3b6a92ce with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 71bf5a94c99d18879b298e3ef0b271a0 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/2bc1ddab-71b1-4119-b213-94c147acb2e8_calibrate_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/2bc1ddab-71b1-4119-b213-94c147acb2e8_calibrate_1123954_5.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2b52c81e-4f35-4a7a-866c-a135ceef91e9_characterizeImage_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2b52c81e-4f35-4a7a-866c-a135ceef91e9_characterizeImage_1123954_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 73 try 1 failed parsl.dataflow.dflow ERROR: Task 73 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c61a15b407efe1a18fa6bdaf28706531 with result from task 73 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 74 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 74 try 0 failed parsl.dataflow.dflow INFO: Task 74 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b081d7eb-49ac-47eb-9e42-a64bc031b8a6_calibrate_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b081d7eb-49ac-47eb-9e42-a64bc031b8a6_calibrate_1124087_21.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/f68c57f5-9001-4e10-b974-94e1c5400265_characterizeImage_1124087_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:06 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:06 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:06 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154021f8851b3fa04995b32ae727503d with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 07974d7b11152256b90954224d50e988 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/de7ffc46-9b78-4d63-ad15-615909fc1767_calibrate_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/de7ffc46-9b78-4d63-ad15-615909fc1767_calibrate_1123954_52.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9db5c473-f69f-47d2-b332-4b46ba0b6104_characterizeImage_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9db5c473-f69f-47d2-b332-4b46ba0b6104_characterizeImage_1123954_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:07 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4813cf1b43fae9bd49dc6d620e73480 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 53d3068aca78df540e7220f305945397 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5249c515-ffc9-484c-988a-96e480778854_calibrate_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5249c515-ffc9-484c-988a-96e480778854_calibrate_1123958_24.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8b9c434e-94d0-4a5b-ac4b-d34485f1fa2c_characterizeImage_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8b9c434e-94d0-4a5b-ac4b-d34485f1fa2c_characterizeImage_1123958_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:07 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 1 failed parsl.dataflow.dflow ERROR: Task 124 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8eab558966ecefd80044f943556fe898 with result from task 124 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 125 try 0 failed parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b6096f19-1cd2-43a1-b4dc-89d1d7f7bfd8_calibrate_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b6096f19-1cd2-43a1-b4dc-89d1d7f7bfd8_calibrate_1124087_7.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/0389377c-7cac-4428-a369-5dfb8c6ec2e7_characterizeImage_1124087_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:08 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 1 failed parsl.dataflow.dflow ERROR: Task 118 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 03c5fa55de0adea66ef513e35166dc4c with result from task 118 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 119 try 0 failed parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/3f80fa96-1687-4c90-be48-2c40e6bdc263_calibrate_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/3f80fa96-1687-4c90-be48-2c40e6bdc263_calibrate_1124087_47.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/512e7973-06b1-468a-9878-d3e7f3d158a8_characterizeImage_1124087_47.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 522b25a7f14981f2d6a14d6f8c08009a with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash ea77285ca7c6387f26fe98651d636b85 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e55a5b10-71db-40ea-9908-03190d54d021_calibrate_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e55a5b10-71db-40ea-9908-03190d54d021_calibrate_1123956_15.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a2b6e6c7-b247-4014-a376-ab4febdff69a_characterizeImage_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a2b6e6c7-b247-4014-a376-ab4febdff69a_characterizeImage_1123956_15.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 25 try 1 failed parsl.dataflow.dflow ERROR: Task 25 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 81f87314cd2d2eea05ca4fe6ba13d991 with result from task 25 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 26 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 26 try 0 failed parsl.dataflow.dflow INFO: Task 26 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/d19847f5-ff3c-4118-aedb-d8125f60178f_calibrate_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/d19847f5-ff3c-4118-aedb-d8125f60178f_calibrate_1124087_3.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1d0f3fa1-e048-4b67-9ee8-597b4ac82faa_characterizeImage_1124087_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 229 parsl.dataflow.strategy DEBUG: Executor multi has 229 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 229 parsl.dataflow.strategy DEBUG: Executor multi has 229 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85e9ad05348ce00d30b61fbf3e6c1612 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 004fae222d149b5d5f2007975fbaba87 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/58c40b57-818d-49a7-8440-bfcae84b7d04_calibrate_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/58c40b57-818d-49a7-8440-bfcae84b7d04_calibrate_1123958_7.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3a44df86-a44c-4e5c-b49f-1d55ec37a8da_characterizeImage_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3a44df86-a44c-4e5c-b49f-1d55ec37a8da_characterizeImage_1123958_7.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86e793ca89fa88f4ab668734bbdfcbd3 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash f4ecc67abe9852bdff537342ef578d70 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c173240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0da6ab02-b884-49ce-a19d-cd402a6658df_calibrate_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0da6ab02-b884-49ce-a19d-cd402a6658df_calibrate_1123954_57.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1115a916-728f-4eb8-875d-742548fc925c_characterizeImage_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1115a916-728f-4eb8-875d-742548fc925c_characterizeImage_1123954_57.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 1 failed parsl.dataflow.dflow ERROR: Task 76 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec8a93091cc7028539489f31e6eae2a with result from task 76 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 77 try 0 failed parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ad08a14a-bf34-4611-be17-0265eaf17e93_calibrate_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ad08a14a-bf34-4611-be17-0265eaf17e93_calibrate_1124087_33.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9cb9e11d-3014-43b0-8fc4-c872185ac393_characterizeImage_1124087_33.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 334 try 1 failed parsl.dataflow.dflow ERROR: Task 334 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3ff3465232b98d1e68709757d353b81a with result from task 334 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 335 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 335 try 0 failed parsl.dataflow.dflow INFO: Task 335 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/839b50f4-bb17-42d6-a1d0-19a6b3e920f5_calibrate_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/839b50f4-bb17-42d6-a1d0-19a6b3e920f5_calibrate_1124087_32.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/87002d80-a465-43ce-b20a-8d32f484996a_characterizeImage_1124087_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 418 try 1 failed parsl.dataflow.dflow ERROR: Task 418 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry db114faafb5b568b7d87f36f26b1c706 with result from task 418 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 419 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 419 try 0 failed parsl.dataflow.dflow INFO: Task 419 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7c624edb-6ce2-4af9-a27f-c80f67b3ae7c_calibrate_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7c624edb-6ce2-4af9-a27f-c80f67b3ae7c_calibrate_1124087_28.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e35a204a-bf01-4f96-adcd-cee7d999a639_characterizeImage_1124087_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 205 try 1 failed parsl.dataflow.dflow ERROR: Task 205 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 202f87230bbf2c309e5edc9856b456e2 with result from task 205 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 206 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 206 try 0 failed parsl.dataflow.dflow INFO: Task 206 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/fb8900d4-2b3c-4ce8-9cbb-a6be331a27c5_calibrate_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/fb8900d4-2b3c-4ce8-9cbb-a6be331a27c5_calibrate_1124087_6.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5edfda8d-059c-4341-a384-2117852e38f7_characterizeImage_1124087_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 55 try 1 failed parsl.dataflow.dflow ERROR: Task 55 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 553470b5f4d9ae60e6df7935880cea98 with result from task 55 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 56 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 56 try 0 failed parsl.dataflow.dflow INFO: Task 56 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ae8434ab-c01c-4389-9fb8-e1059bbd152a_calibrate_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ae8434ab-c01c-4389-9fb8-e1059bbd152a_calibrate_1124087_40.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/6a541ae6-8c21-4c46-9b01-cd611f7f4698_characterizeImage_1124087_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 319 try 1 failed parsl.dataflow.dflow ERROR: Task 319 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 487f3bb7d6d1e9c23d93b3583c090732 with result from task 319 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 320 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 320 try 0 failed parsl.dataflow.dflow INFO: Task 320 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e8ae5ac2-7b9b-4b1b-9be6-66850230fb8f_calibrate_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e8ae5ac2-7b9b-4b1b-9be6-66850230fb8f_calibrate_1124087_50.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/54338598-0a26-4bca-bb8a-a106808071ec_characterizeImage_1124087_50.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 223 parsl.dataflow.strategy DEBUG: Executor multi has 223 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 331 try 1 failed parsl.dataflow.dflow ERROR: Task 331 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9e58bffff67bb791c13924d14b4ae5d5 with result from task 331 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 332 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 332 try 0 failed parsl.dataflow.dflow INFO: Task 332 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/337a2686-3d9b-4547-83b7-9e72a7d37125_calibrate_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/337a2686-3d9b-4547-83b7-9e72a7d37125_calibrate_1124087_56.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ba2fdb4c-0832-4a98-9fac-d55d0a0316a2_characterizeImage_1124087_56.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 412 try 1 failed parsl.dataflow.dflow ERROR: Task 412 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bae99d87cdf984b44621a63899e76f3e with result from task 412 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 413 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 413 try 0 failed parsl.dataflow.dflow INFO: Task 413 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/10709412-c5e2-48cf-9647-e12480224d79_calibrate_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/10709412-c5e2-48cf-9647-e12480224d79_calibrate_1124087_55.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a94ab299-5de4-4120-a4ee-dd75eda6b2da_characterizeImage_1124087_55.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 415 try 1 failed parsl.dataflow.dflow ERROR: Task 415 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5077b83a1c2ad93fb9e16caedfcd81e3 with result from task 415 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 416 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 416 try 0 failed parsl.dataflow.dflow INFO: Task 416 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/a6a25ae1-4dff-4bf2-98b2-c92a8f92db42_calibrate_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/a6a25ae1-4dff-4bf2-98b2-c92a8f92db42_calibrate_1124087_29.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e3c24664-bc51-4169-bbbd-509f148b211d_characterizeImage_1124087_29.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 292 try 1 failed parsl.dataflow.dflow ERROR: Task 292 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9b11d8c391f1add22d2e21dd6bc55f26 with result from task 292 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 293 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 293 try 0 failed parsl.dataflow.dflow INFO: Task 293 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/d60aace1-5fb3-4d21-8250-cf072e78c002_calibrate_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/d60aace1-5fb3-4d21-8250-cf072e78c002_calibrate_1124087_30.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/a196f296-98e7-4076-8941-9538af614527_characterizeImage_1124087_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 265 try 1 failed parsl.dataflow.dflow ERROR: Task 265 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 358b9ee01929f6a95a8ea19df9a5f947 with result from task 265 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 266 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 266 try 0 failed parsl.dataflow.dflow INFO: Task 266 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/8511b245-a569-4d1f-bff7-55cdaa2a34e6_calibrate_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/8511b245-a569-4d1f-bff7-55cdaa2a34e6_calibrate_1124087_51.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d702f1f9-4374-4ad5-bc03-2986b1dea403_characterizeImage_1124087_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 1 failed parsl.dataflow.dflow ERROR: Task 1 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c000c493180d38cfca78582ff6fe4f13 with result from task 1 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 2 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 2 try 0 failed parsl.dataflow.dflow INFO: Task 2 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/00e6fd48-29d9-4299-b680-eb4c907c7317_calibrate_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/00e6fd48-29d9-4299-b680-eb4c907c7317_calibrate_1124087_18.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/64ad14e3-4f00-48cc-90cd-64cb6332dea9_characterizeImage_1124087_18.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dc484c3c81d180d4ede135c752173ef with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash af76411577c0c46d8efa546d33016b09 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca57560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9e53aea9-168b-476d-92ab-a8fed3fe3aa2_calibrate_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9e53aea9-168b-476d-92ab-a8fed3fe3aa2_calibrate_1123958_20.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/9aa81ef7-e9ef-4eb9-9813-036a4592c2fd_characterizeImage_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/9aa81ef7-e9ef-4eb9-9813-036a4592c2fd_characterizeImage_1123958_20.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbaf790f2318605eb2edfcb2e01375ee with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 94baa9fdf30bdc8c9cfa57937d3a08c2 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e3d8aac8-b3b7-4883-848e-c79b8c82dc59_calibrate_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e3d8aac8-b3b7-4883-848e-c79b8c82dc59_calibrate_1123954_45.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1866feac-2038-44d4-89b4-da7c7dc92298_characterizeImage_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1866feac-2038-44d4-89b4-da7c7dc92298_characterizeImage_1123954_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 742 try 0 failed parsl.dataflow.dflow INFO: Task 742 marked for retry parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash efa1be6b475a8acc5557333513890141 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688721a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 1 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 172 try 0 failed parsl.dataflow.dflow INFO: Task 172 marked for retry parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 6834881ba9fc070da018d95407d6c1d9 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 1 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 445 try 1 failed parsl.dataflow.dflow ERROR: Task 445 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f503c85b76021efdc5f2bcc713bdc08d with result from task 445 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 446 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 446 try 0 failed parsl.dataflow.dflow INFO: Task 446 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e180049a-7fee-4854-8d4d-10ba52ce5b17_calibrate_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e180049a-7fee-4854-8d4d-10ba52ce5b17_calibrate_1124087_13.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/90dd3efc-c568-4ac2-adc0-5ef7e5ba91b2_characterizeImage_1124087_13.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 653186249736e1e9b4602f84088c307b with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash e710048414e777f9fc0342a5e947df88 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/56e3f47c-b515-456a-8d0e-39487889a2aa_calibrate_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/56e3f47c-b515-456a-8d0e-39487889a2aa_calibrate_1123958_48.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/812ca4b8-8327-4ffe-9450-1dce5c7c52cf_characterizeImage_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/812ca4b8-8327-4ffe-9450-1dce5c7c52cf_characterizeImage_1123958_48.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb6f2a67c66028e158ccfec1bcec9f0f with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 72a77d83b0f53e140422d048d358eeb8 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/20ede348-3df5-414a-82c2-9972dd218fea_calibrate_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/20ede348-3df5-414a-82c2-9972dd218fea_calibrate_1123958_62.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/186e0916-aef7-454a-8628-5672a18d8246_characterizeImage_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/186e0916-aef7-454a-8628-5672a18d8246_characterizeImage_1123958_62.stderr 2024-03-07 03:41:18 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:41:18 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:18 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 241 try 1 failed parsl.dataflow.dflow ERROR: Task 241 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 077a92be8651eb7013542b5d4fdebe42 with result from task 241 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 242 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 242 try 0 failed parsl.dataflow.dflow INFO: Task 242 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/4538b2ff-cd1a-406c-869f-9b66b104bf8e_calibrate_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/4538b2ff-cd1a-406c-869f-9b66b104bf8e_calibrate_1124087_16.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/52267d35-c0a4-4f53-8f52-0c4663d5c2a2_characterizeImage_1124087_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:19 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71e391097c53ed91a2fb1d2652808654 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash dd32498e4d96904e141c4b949bf281de parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/eb92e0e6-59e0-43e5-b892-6d80e468e291_calibrate_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/eb92e0e6-59e0-43e5-b892-6d80e468e291_calibrate_1123954_55.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/afe6afee-437a-4513-a699-18c943ce7488_characterizeImage_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/afe6afee-437a-4513-a699-18c943ce7488_characterizeImage_1123954_55.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 301 try 1 failed parsl.dataflow.dflow ERROR: Task 301 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dd4a8f92fa928307b9e41bd81bd2bdfe with result from task 301 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 302 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 302 try 0 failed parsl.dataflow.dflow INFO: Task 302 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/494c29b7-a040-4608-b167-e13d4903ded6_calibrate_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/494c29b7-a040-4608-b167-e13d4903ded6_calibrate_1124087_14.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d9aca372-3dcb-41ec-9d9d-7537d2ce986b_characterizeImage_1124087_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 367 try 0 failed parsl.dataflow.dflow INFO: Task 367 marked for retry parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 5b350e802da89d3a6ed368f12907824b parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 1 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:19 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f40169e9adde144ad582596f9dbb4e5 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 81ebc3d86a9cf2b9e158aca61b1b7429 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/330a729b-5f3d-4254-9844-206afc6b006a_calibrate_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/330a729b-5f3d-4254-9844-206afc6b006a_calibrate_1123954_4.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/476f4296-62c3-4c9d-b32d-b3fc7e421357_characterizeImage_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/476f4296-62c3-4c9d-b32d-b3fc7e421357_characterizeImage_1123954_4.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4212e9f97d2818b9b2966fd4489612d with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 2316d1f639034ed1bac079f2d51c10d4 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f6544871-761f-416e-9e94-b04155f34cb8_calibrate_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f6544871-761f-416e-9e94-b04155f34cb8_calibrate_1123954_38.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2385db8e-8dab-458c-a821-c26c96228b4d_characterizeImage_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/2385db8e-8dab-458c-a821-c26c96228b4d_characterizeImage_1123954_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:19 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:20 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:20 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:21 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:21 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 358 try 0 failed parsl.dataflow.dflow INFO: Task 358 marked for retry parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 6285ea3f51196c4bee54f7e2ecf04112 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 1 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:21 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:22 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 373 try 0 failed parsl.dataflow.dflow INFO: Task 373 marked for retry parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash a7bb50e5d8074ea951d1cf4dc9701aa5 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 1 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee496ae044b900e149a690bbd4b5824 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 2dd043c7b1d839d4e82f377016d3125d parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b8481e0c-19d9-4438-b890-c8b6bdaa0646_calibrate_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b8481e0c-19d9-4438-b890-c8b6bdaa0646_calibrate_1123958_53.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/371666f9-0f49-44fd-8a9a-1fcc99da5cd0_characterizeImage_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/371666f9-0f49-44fd-8a9a-1fcc99da5cd0_characterizeImage_1123958_53.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b0f1ad1a11e37b5a16f45a961e35507 with result from task 481 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:22 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 08a38ee395e12e786e84f38c54bc649e parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/db3d3865-ea93-4fe3-913a-fcbe9294d1c0_calibrate_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/db3d3865-ea93-4fe3-913a-fcbe9294d1c0_calibrate_1123956_18.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d213ad62-b453-47e6-8001-9a217d76f0c8_characterizeImage_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d213ad62-b453-47e6-8001-9a217d76f0c8_characterizeImage_1123956_18.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 364 try 0 failed parsl.dataflow.dflow INFO: Task 364 marked for retry parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 971fa46d26c1ddde2f6c5eab813ba3e4 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 1 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:22 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:23 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 232 try 1 failed parsl.dataflow.dflow ERROR: Task 232 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7e28b64007360f68cbc861ee13394355 with result from task 232 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 233 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 233 try 0 failed parsl.dataflow.dflow INFO: Task 233 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b0632a43-87eb-4994-af4f-2f1a7d106164_calibrate_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b0632a43-87eb-4994-af4f-2f1a7d106164_calibrate_1124087_60.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/77b64d5c-3e68-4e0a-8dbb-e81a26311ba4_characterizeImage_1124087_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:23 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:23 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:23 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 247 try 1 failed parsl.dataflow.dflow ERROR: Task 247 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7bbf191a2c92df9540b1e43f7a147f5a with result from task 247 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 248 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 248 try 0 failed parsl.dataflow.dflow INFO: Task 248 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e9b9e362-2f5b-4107-8218-5e80939a8219_calibrate_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e9b9e362-2f5b-4107-8218-5e80939a8219_calibrate_1124087_24.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1888f9d8-c26c-4d93-9a63-b4fb04b4c017_characterizeImage_1124087_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:24 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cdbfce54988d152a0bd842ddb511bd1 with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash fe023f772bc85058019f3d60ff704fb5 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b59cf040-ea1e-4d16-9d03-ee025bc681c3_calibrate_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b59cf040-ea1e-4d16-9d03-ee025bc681c3_calibrate_1123956_50.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/17828148-447b-49bd-8197-9a4c9a11159b_characterizeImage_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/17828148-447b-49bd-8197-9a4c9a11159b_characterizeImage_1123956_50.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4461331b207912c314c88448f3074791 with result from task 511 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:24 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 0902645de332b05718e9eb202b24ccf3 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8d2b3b41-6ba8-4210-a4ef-dd99150c93cd_calibrate_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8d2b3b41-6ba8-4210-a4ef-dd99150c93cd_calibrate_1123958_43.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8263b61f-4011-4957-abc8-3af7883f9ada_characterizeImage_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8263b61f-4011-4957-abc8-3af7883f9ada_characterizeImage_1123958_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:25 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:25 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b37c1a64d9306da0b88b57e73adbec4a with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 5ec838f5399fa0bc0aaec9673f576de5 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a9ef69ba-9517-4aa3-9eac-823457fca4a7_calibrate_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a9ef69ba-9517-4aa3-9eac-823457fca4a7_calibrate_1123954_20.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/16037a06-1ab0-46b8-8dec-51c557cd0566_characterizeImage_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/16037a06-1ab0-46b8-8dec-51c557cd0566_characterizeImage_1123954_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:25 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:26 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:26 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:27 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:27 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:27 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1f8917c8f20024e36ad4c00c5581a71 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 7d031d13835f8958a0fb3b15617ff683 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ef550ae2-9ed4-4a4c-9f53-3d13427cd12d_calibrate_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ef550ae2-9ed4-4a4c-9f53-3d13427cd12d_calibrate_1123956_49.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/f21f079d-efdc-41db-a473-72895056d33d_characterizeImage_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/f21f079d-efdc-41db-a473-72895056d33d_characterizeImage_1123956_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:28 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e70676d234d2e724a0549b8244a8ef97 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash a2b6e81e1bb84e8e876d346e819cb7f9 parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/81e2d5fa-3c53-4669-af47-045ea748ab13_calibrate_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/81e2d5fa-3c53-4669-af47-045ea748ab13_calibrate_1123956_43.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/930d4d3f-af55-435e-b76a-cc9da56844e9_characterizeImage_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/930d4d3f-af55-435e-b76a-cc9da56844e9_characterizeImage_1123956_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:28 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 466 try 1 failed parsl.dataflow.dflow ERROR: Task 466 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 83436b1bb1675282fef2ea3ec779392c with result from task 466 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 467 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 467 try 0 failed parsl.dataflow.dflow INFO: Task 467 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/13234be9-7ef3-4007-8973-fe781267366b_calibrate_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/13234be9-7ef3-4007-8973-fe781267366b_calibrate_1124087_57.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/9b3dbe54-91f3-4ea6-912f-310ac983004e_characterizeImage_1124087_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:28 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:29 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:29 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:29 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:30 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:30 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:31 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:31 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:31 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 490 try 1 failed parsl.dataflow.dflow ERROR: Task 490 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 363b64fdab6918421123bfca0735f94a with result from task 490 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 491 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 491 try 0 failed parsl.dataflow.dflow INFO: Task 491 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/75742421-df35-478c-8dc8-51260c86f041_calibrate_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/75742421-df35-478c-8dc8-51260c86f041_calibrate_1124087_48.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/dfc9d79b-a2f9-41cf-9071-bb9c75205fe8_characterizeImage_1124087_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:32 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:32 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45cc0282848d7c3f11b3f77e8d5724f0 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 1e40ead427df9d7ae3b8442f5f06d25e parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c172f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8ae4d37c-57d1-481b-a107-4ff840c3d6b1_calibrate_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8ae4d37c-57d1-481b-a107-4ff840c3d6b1_calibrate_1123954_39.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/83425bc8-c9a6-4ca7-af7b-5ae323e82629_characterizeImage_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/83425bc8-c9a6-4ca7-af7b-5ae323e82629_characterizeImage_1123954_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:32 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:33 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b412f28916c5a7107865ca67bfeb661 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 8df78e5dc09e6f4cef3b07a72799530d parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4309fd8d-c694-4775-8b65-3a585a1fc71e_calibrate_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4309fd8d-c694-4775-8b65-3a585a1fc71e_calibrate_1123956_20.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/acf5759c-1d87-4987-a51e-c96c09bdf6cc_characterizeImage_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/acf5759c-1d87-4987-a51e-c96c09bdf6cc_characterizeImage_1123956_20.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b6674ebcadc99ccee11511c2916929f with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 73f85a4a27ff2a45b01a2b5d010a0511 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1728e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/39d257ef-e5ce-4a69-8665-2eecd2c5d6df_calibrate_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/39d257ef-e5ce-4a69-8665-2eecd2c5d6df_calibrate_1123956_51.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/af8fb548-3425-4acb-9131-50b95a81c97f_characterizeImage_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/af8fb548-3425-4acb-9131-50b95a81c97f_characterizeImage_1123956_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:33 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:34 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:34 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 472 try 1 failed parsl.dataflow.dflow ERROR: Task 472 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c66aaadbf21907522ac40b342a1e59a8 with result from task 472 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 473 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 473 try 0 failed parsl.dataflow.dflow INFO: Task 473 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/15c31e31-ba60-4ea7-93e0-e42282b2f4be_calibrate_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/15c31e31-ba60-4ea7-93e0-e42282b2f4be_calibrate_1124087_12.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/edef6094-0d0d-4f4b-87bc-96f1704aa79a_characterizeImage_1124087_12.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 546d9fbdd23feec01b2449ccb2579898 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 2c45314f21d8259322d595e77ca19ea3 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/a9ab9d0c-5061-4760-9f5e-2d379ddd0f6f_calibrate_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/a9ab9d0c-5061-4760-9f5e-2d379ddd0f6f_calibrate_1123956_5.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a19b023c-349d-4113-b897-4e66c2c9ee01_characterizeImage_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/a19b023c-349d-4113-b897-4e66c2c9ee01_characterizeImage_1123956_5.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0d16f8a79cb1bab18197620f17f5ab8 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 487a1cef26bbd096a39924d047732bcd parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5b7325bf-2e05-454f-bd88-bc4ed221f128_calibrate_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5b7325bf-2e05-454f-bd88-bc4ed221f128_calibrate_1123956_1.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/790fa9e4-d5cc-44ae-b9de-882af8d655ac_characterizeImage_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/790fa9e4-d5cc-44ae-b9de-882af8d655ac_characterizeImage_1123956_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:36 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:36 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:37 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:37 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:37 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:38 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:38 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:38 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c84594cee11d6839a3da1a3596266118 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash d335a52c2ee64fd7dcb232063440595f parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873249a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/798426ff-055b-431d-890a-125a32f99de6_calibrate_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/798426ff-055b-431d-890a-125a32f99de6_calibrate_1123954_15.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3813206f-9bd5-4946-985e-c3741133ccbd_characterizeImage_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3813206f-9bd5-4946-985e-c3741133ccbd_characterizeImage_1123954_15.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 743828eba8e08f7a55bf571ac6cc0b92 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 256bbe1f8521e4141239fdafbb273dea parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1296c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d22b666f-da5a-42dd-be7d-f059c1d5c5fb_calibrate_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d22b666f-da5a-42dd-be7d-f059c1d5c5fb_calibrate_1123954_60.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0a1f55c9-cb9d-407d-93fd-ddbed6aac68c_characterizeImage_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/0a1f55c9-cb9d-407d-93fd-ddbed6aac68c_characterizeImage_1123954_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:39 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 918bb1c25e966d3ba996ca329e594e1b with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 781753c15bcdd8eceb2640d88d0fad29 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c170f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8642def3-6bfb-49c9-bd0e-6d85d5a5450e_calibrate_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8642def3-6bfb-49c9-bd0e-6d85d5a5450e_calibrate_1123954_27.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/8dd71392-b0e4-4923-a96e-2bb9820dbacd_characterizeImage_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/8dd71392-b0e4-4923-a96e-2bb9820dbacd_characterizeImage_1123954_27.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38e1729962ccc5d0a83f49939d36f56d with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash f84a0f8ac9b6fba5754888552c0285b7 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/0370d87b-8238-484e-9b01-cb0abbde6feb_calibrate_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/0370d87b-8238-484e-9b01-cb0abbde6feb_calibrate_1123958_41.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/782c6997-0b79-437d-9425-314e2a3bcfeb_characterizeImage_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/782c6997-0b79-437d-9425-314e2a3bcfeb_characterizeImage_1123958_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:39 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:39 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50a718b03053643f427761329f61e9e7 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 1077cff816795d689032ffa4378ce73a parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/23ac7a5e-f363-459a-94e1-7c27611831a6_calibrate_1123956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/23ac7a5e-f363-459a-94e1-7c27611831a6_calibrate_1123956_55.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b51d1054-8efb-4dc0-bf74-c1a18341bee8_characterizeImage_1123956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/b51d1054-8efb-4dc0-bf74-c1a18341bee8_characterizeImage_1123956_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:40 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:40 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 487 try 1 failed parsl.dataflow.dflow ERROR: Task 487 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 47f4d4447ab1302d3ca6abd182923b02 with result from task 487 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 488 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 488 try 0 failed parsl.dataflow.dflow INFO: Task 488 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/54a60845-77d2-47e8-8b1a-660030a83273_calibrate_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/54a60845-77d2-47e8-8b1a-660030a83273_calibrate_1124087_17.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/478f13b8-b8af-4c96-9ca6-e19710809957_characterizeImage_1124087_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:41 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:41 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9c00af72112cd6ea695962e424c1a54 with result from task 691 parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:41 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 3f8b191a99d71ab0a5de1cc728ecf059 parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6cf0bab4-752b-4689-bc35-d04032699e59_calibrate_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6cf0bab4-752b-4689-bc35-d04032699e59_calibrate_1123958_40.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b9ff879a-1759-4b29-8b51-da8b13f22e6c_characterizeImage_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b9ff879a-1759-4b29-8b51-da8b13f22e6c_characterizeImage_1123958_40.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d17cb0893c545c8b45d1d23120a82db8 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash d5000c0f68db8e2fe6b223c079b40f4e parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/41f9f517-aa10-4afa-8b42-7e22155d1a4c_calibrate_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/41f9f517-aa10-4afa-8b42-7e22155d1a4c_calibrate_1123954_62.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f22306b3-49d2-47cc-916b-572f1eee7d9b_characterizeImage_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f22306b3-49d2-47cc-916b-572f1eee7d9b_characterizeImage_1123954_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 208 parsl.dataflow.strategy DEBUG: Executor multi has 208 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 208 parsl.dataflow.strategy DEBUG: Executor multi has 208 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09c18a2a96187721ccd69fe9bbc484a7 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash c8d5a24c4753d80b87dca7c17bd4ba8e parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/9c11d566-09ec-4ed5-b9bc-3cc7a808d330_calibrate_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/9c11d566-09ec-4ed5-b9bc-3cc7a808d330_calibrate_1123954_28.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/98cf117a-5151-415f-b174-9c48dd8c31c2_characterizeImage_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/98cf117a-5151-415f-b174-9c48dd8c31c2_characterizeImage_1123954_28.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41858345246c09e368d8662a258c5c19 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 2d4684bf8c9dbbe14c697a42701c263d parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/1c366220-030f-4fb4-905c-a7b0d8d3366b_calibrate_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/1c366220-030f-4fb4-905c-a7b0d8d3366b_calibrate_1123954_12.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3c6c2c9f-e427-4c17-a2b8-31529977ce0b_characterizeImage_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3c6c2c9f-e427-4c17-a2b8-31529977ce0b_characterizeImage_1123954_12.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f5748f4932d4a5db24e745cb024211 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 3f95e10a93188269542786cbef826b28 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/abe05d06-17ff-4b9d-a486-b40780616425_calibrate_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/abe05d06-17ff-4b9d-a486-b40780616425_calibrate_1123956_39.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d4fd0b0e-c4ef-4fea-ae72-ad782d130769_characterizeImage_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/d4fd0b0e-c4ef-4fea-ae72-ad782d130769_characterizeImage_1123956_39.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 323 try 0 failed parsl.dataflow.dflow INFO: Task 323 marked for retry parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 91ad19b921b3396663b2b24ea71d25d6 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 1 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46054bda177d3fba5c4f772aac4763fa with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash eb6b1134157037d459dd836fdc02977a parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/063c2b52-5bab-4458-b4e9-012292f5f426_calibrate_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/063c2b52-5bab-4458-b4e9-012292f5f426_calibrate_1123956_23.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ea917666-fd94-4d21-b963-a3193548b1da_characterizeImage_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ea917666-fd94-4d21-b963-a3193548b1da_characterizeImage_1123956_23.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 529 try 1 failed parsl.dataflow.dflow ERROR: Task 529 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8970d5e191d44c57151af1768dd92906 with result from task 529 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 530 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 530 try 0 failed parsl.dataflow.dflow INFO: Task 530 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ff637001-5324-46eb-a19a-f79f1a0b74aa_calibrate_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/ff637001-5324-46eb-a19a-f79f1a0b74aa_calibrate_1124087_58.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/7d6fcf6e-d6ae-4282-9f15-7b47916c1610_characterizeImage_1124087_58.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 574 try 1 failed parsl.dataflow.dflow ERROR: Task 574 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3fb2648f636aa96f82dd546aee16de2e with result from task 574 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 575 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 575 try 0 failed parsl.dataflow.dflow INFO: Task 575 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/3297d840-e072-469d-a14c-924bb76dddde_calibrate_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/3297d840-e072-469d-a14c-924bb76dddde_calibrate_1124087_44.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5369d2bb-040f-47e0-968e-495cb07b2a51_characterizeImage_1124087_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb2d29aa89135698a92ce79ef4b80787 with result from task 544 parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 205 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 875e9b762e93e5e39be33376e09bf4a5 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887325620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/e64be338-85cc-490b-b8c2-f0831ae1019b_calibrate_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/e64be338-85cc-490b-b8c2-f0831ae1019b_calibrate_1123958_52.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/047ccf90-259e-4689-a70e-493c0c51ae43_characterizeImage_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/047ccf90-259e-4689-a70e-493c0c51ae43_characterizeImage_1123958_52.stderr parsl.dataflow.strategy DEBUG: Executor multi has 205 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 758283cbaa3d600bae5289a3aeea992d with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 284d0f80e86ba29f5c050fcbb29e2d99 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1440bc28-cffe-4c86-ba15-14355f359f0c_calibrate_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1440bc28-cffe-4c86-ba15-14355f359f0c_calibrate_1123958_32.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ea566331-cf92-43e9-b607-7059131152dc_characterizeImage_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/ea566331-cf92-43e9-b607-7059131152dc_characterizeImage_1123958_32.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eacfe80d34dee1131af6ce38b1c1e0fd with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash a64b9edcb64bf228ba2514a243e076c1 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/04ea1439-5700-4d64-a576-e4dccf41dc95_calibrate_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/04ea1439-5700-4d64-a576-e4dccf41dc95_calibrate_1123958_47.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e6d96f7c-d0ec-4e7b-818e-ebe30d5517e2_characterizeImage_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e6d96f7c-d0ec-4e7b-818e-ebe30d5517e2_characterizeImage_1123958_47.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 688 try 1 failed parsl.dataflow.dflow ERROR: Task 688 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5e196acf326a5f34a6381ef10c3f3581 with result from task 688 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 689 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 689 try 0 failed parsl.dataflow.dflow INFO: Task 689 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f44ce84f-4d38-4a74-92e1-37e5eeddf11f_calibrate_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f44ce84f-4d38-4a74-92e1-37e5eeddf11f_calibrate_1123958_2.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/b1616998-da5a-4338-9c53-f0efd882da5c_characterizeImage_1123958_2.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e713d09849d31888595fbba3df09f82 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 80050303745a0f4b56730859d6c09cab parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5d44d633-184e-4e0a-b259-f1f92c5936db_calibrate_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5d44d633-184e-4e0a-b259-f1f92c5936db_calibrate_1123958_16.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d5d926aa-06db-498a-af81-2a2f86d44cb4_characterizeImage_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/d5d926aa-06db-498a-af81-2a2f86d44cb4_characterizeImage_1123958_16.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a5be7c5d04de8bfa4fa4662a8ca6e8 with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 6adaba0208f40f3a7ca3143ec9998667 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f2f0b687-9c49-4b6f-a367-8dcfe92744d4_calibrate_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f2f0b687-9c49-4b6f-a367-8dcfe92744d4_calibrate_1123956_16.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7283b45e-9eae-4036-b98b-d90206f5e883_characterizeImage_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/7283b45e-9eae-4036-b98b-d90206f5e883_characterizeImage_1123956_16.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd16c8c59b942bbe064c9eb2e7845f80 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash c0af5a77a148c76a9a145f7e570a5f1e parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1705e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/14430dba-2165-4fd2-931d-12e2e1001bb0_calibrate_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/14430dba-2165-4fd2-931d-12e2e1001bb0_calibrate_1123956_24.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e42a471-1c74-4c93-8f23-e2f5db1d8b70_characterizeImage_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e42a471-1c74-4c93-8f23-e2f5db1d8b70_characterizeImage_1123956_24.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 589 try 1 failed parsl.dataflow.dflow ERROR: Task 589 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ce6b71be7acce86cecfcca01dde72da0 with result from task 589 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 590 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 590 try 0 failed parsl.dataflow.dflow INFO: Task 590 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8015a330-019e-4f7a-80df-06b2a2b374d9_calibrate_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8015a330-019e-4f7a-80df-06b2a2b374d9_calibrate_1123954_2.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a14e28b1-80fa-4b59-b723-d984c60a81bf_characterizeImage_1123954_2.stderr 2024-03-07 03:41:52 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:41:52 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:52 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:53 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 963718ed5a97b3a561d403e9196ac271 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash af79b4a361af586aa51ac71474fcc55c parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a60e78b-1786-46e1-9b1a-1d3cb4f16bae_calibrate_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a60e78b-1786-46e1-9b1a-1d3cb4f16bae_calibrate_1123956_48.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/bb9d4330-ebea-47ab-88e8-6698eff01bb0_characterizeImage_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/bb9d4330-ebea-47ab-88e8-6698eff01bb0_characterizeImage_1123956_48.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 613 try 1 failed parsl.dataflow.dflow ERROR: Task 613 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 24b5f9a1817d60ccca15f2d84dd4de95 with result from task 613 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 614 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 614 try 0 failed parsl.dataflow.dflow INFO: Task 614 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7ca5e91b-5429-48d7-bd52-78f0c8c59e57_calibrate_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7ca5e91b-5429-48d7-bd52-78f0c8c59e57_calibrate_1124087_49.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/93dfd29c-40bd-47c0-856e-abc1e5dd15d5_characterizeImage_1124087_49.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b25045ae5b1937f3efdf07aebe629c8 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 624f0598a4b4bf88a7d24d650a2181a7 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/74dfebc2-094d-4f78-8084-ade00f89579e_calibrate_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/74dfebc2-094d-4f78-8084-ade00f89579e_calibrate_1123954_35.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7d97295a-da9f-446f-b72e-8d1fa7679303_characterizeImage_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7d97295a-da9f-446f-b72e-8d1fa7679303_characterizeImage_1123954_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:53 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:53 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38b1d10cdf915eec3e4b426d386fcd1f with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 6b937502ae5e6bcc50782580453e41ec parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a3fb4516-6a1c-4a83-af57-effd2eb4f495_calibrate_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a3fb4516-6a1c-4a83-af57-effd2eb4f495_calibrate_1123954_6.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f4296afa-5cea-4841-bc92-b38f58a0b429_characterizeImage_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f4296afa-5cea-4841-bc92-b38f58a0b429_characterizeImage_1123954_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:54 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:54 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2a221d9b713d7c341950195155dfa6e with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 643ed61f90a48802373b85a09c9d8053 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/37fbc880-8d65-4e06-9d6c-9086c90fb905_calibrate_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/37fbc880-8d65-4e06-9d6c-9086c90fb905_calibrate_1123954_22.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7df1f56a-1848-4d9e-86b5-06c1f0bf7ea4_characterizeImage_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/7df1f56a-1848-4d9e-86b5-06c1f0bf7ea4_characterizeImage_1123954_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:54 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36ebb9d3750d315256bcddbf119e1e25 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 4d8798a95e43db3c792da28dde557945 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3df3c3a2-d840-4391-b6d5-3330497a5162_calibrate_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3df3c3a2-d840-4391-b6d5-3330497a5162_calibrate_1123956_47.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/98d55b79-8a9a-467d-a0d3-6969886b85f3_characterizeImage_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/98d55b79-8a9a-467d-a0d3-6969886b85f3_characterizeImage_1123956_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:55 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:55 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:56 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 697 try 1 failed parsl.dataflow.dflow ERROR: Task 697 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 15e2b6d411ee96194d43bafeb8a46a56 with result from task 697 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 698 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 698 try 0 failed parsl.dataflow.dflow INFO: Task 698 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/c73f06ee-4514-4151-9739-7f2ce3816db2_calibrate_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/c73f06ee-4514-4151-9739-7f2ce3816db2_calibrate_1124087_19.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb0d2ca-64ab-4a76-bf0b-7c4f82cb3bfb_characterizeImage_1124087_19.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c01704185ff9f50f7e262c655b41039 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 3d487632e1552bf236fef049197abb49 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b713c253-8405-46ef-8f55-b8a1abf48350_calibrate_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b713c253-8405-46ef-8f55-b8a1abf48350_calibrate_1123954_10.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/80cbb4a6-4513-499f-9a46-5aecbc68b98e_characterizeImage_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/80cbb4a6-4513-499f-9a46-5aecbc68b98e_characterizeImage_1123954_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:56 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:56 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:57 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:57 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45022d91f992ecbb1a70939e8d42610f with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 333c48f3f03ce8fdc8384e0858cd6162 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca55940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfb7268e-836e-4d4d-a00c-4a3663626479_calibrate_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfb7268e-836e-4d4d-a00c-4a3663626479_calibrate_1123958_13.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a65da1ef-c0d6-470a-8a08-284a99d406dc_characterizeImage_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a65da1ef-c0d6-470a-8a08-284a99d406dc_characterizeImage_1123958_13.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6be2c6d8428251f30e2527a91e8fe845 with result from task 622 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:57 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 584b259c2a95b0e7ed7b3417a98561a6 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a94281db-1ba1-43ba-b4af-49a5eb87e9a9_calibrate_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a94281db-1ba1-43ba-b4af-49a5eb87e9a9_calibrate_1123954_44.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/926ee7f2-a424-4f11-b7d0-2b8bffa03f35_characterizeImage_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/926ee7f2-a424-4f11-b7d0-2b8bffa03f35_characterizeImage_1123954_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:58 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:58 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:58 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 598 try 1 failed parsl.dataflow.dflow ERROR: Task 598 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 16fdf632038b58740a74668b6c9d0824 with result from task 598 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 599 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 599 try 0 failed parsl.dataflow.dflow INFO: Task 599 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/f6b803a2-3318-4d8f-8cc9-943241c35eb1_calibrate_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/f6b803a2-3318-4d8f-8cc9-943241c35eb1_calibrate_1124087_45.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/d11656ce-c2db-48d2-94dc-c83e5d5cc182_characterizeImage_1124087_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:59 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79c3b04909a45d6b3670ee6885be7d60 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash e755590a82e32f25ea4e5a5cf53ca453 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/89a2bed9-c700-48a4-8d8c-b68fb4b95a2a_calibrate_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/89a2bed9-c700-48a4-8d8c-b68fb4b95a2a_calibrate_1123956_44.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6dec1e93-25db-4189-a884-203088701fe0_characterizeImage_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/6dec1e93-25db-4189-a884-203088701fe0_characterizeImage_1123956_44.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f170f569dee718b14ca2f3e5b8e9f581 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 7549a2e4293c1058bea5b84a4ab852cd parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c1725c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/65ca3b6a-412a-4379-a87b-9c329c4f083a_calibrate_1123956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/65ca3b6a-412a-4379-a87b-9c329c4f083a_calibrate_1123956_36.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0eca7db8-376d-4193-823a-d30412165370_characterizeImage_1123956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/0eca7db8-376d-4193-823a-d30412165370_characterizeImage_1123956_36.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a62c91331f448abc9081d6c1acc3eb33 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 36b6f13e19d2f491442aaefe29487539 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8ab287a1-f86a-4e1a-ab85-cd920376f26e_calibrate_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8ab287a1-f86a-4e1a-ab85-cd920376f26e_calibrate_1123956_9.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8930fe04-c4d0-47eb-8be5-8f68c72e7a1e_characterizeImage_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8930fe04-c4d0-47eb-8be5-8f68c72e7a1e_characterizeImage_1123956_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:59 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09a1934f9a61d193b695598aaf442c32 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 3af96fbb40a505c91c664ab8cba16f37 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/80628c23-fcaa-45c9-a847-4f3e72a58dd6_calibrate_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/80628c23-fcaa-45c9-a847-4f3e72a58dd6_calibrate_1123956_42.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96f93c18-b218-49b4-b63c-05074775da31_characterizeImage_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/96f93c18-b218-49b4-b63c-05074775da31_characterizeImage_1123956_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:41:59 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:00 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7510edeadcbac71713f0550a200eb97 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 22eb6c1ba4d7eea88a1d15a5d192f913 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c129f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/18f627dc-90e1-496a-9534-c6273042fd29_calibrate_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/18f627dc-90e1-496a-9534-c6273042fd29_calibrate_1123954_34.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e56b7148-cdc2-4a43-9758-72f6f75d1610_characterizeImage_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/e56b7148-cdc2-4a43-9758-72f6f75d1610_characterizeImage_1123954_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:00 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 548ebecb202cf1d46aeb628e0b332b13 with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash ee7495069fdf6482ac3708ce4504b6fe parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468872182c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3e661fc1-bf1a-4ae1-ae27-588ed13b01b4_calibrate_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3e661fc1-bf1a-4ae1-ae27-588ed13b01b4_calibrate_1123956_34.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/4f80d198-8ed1-4123-a75d-6dd1b72166fa_characterizeImage_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/4f80d198-8ed1-4123-a75d-6dd1b72166fa_characterizeImage_1123956_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:00 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:01 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:01 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:01 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:02 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:02 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa505f78a961b2d889ac58cdf8197c1e with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash c5cec7fca1931c9df732031c66b0a50c parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/378750f8-83f3-4a1f-97c4-0b4b945bfc8c_calibrate_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/378750f8-83f3-4a1f-97c4-0b4b945bfc8c_calibrate_1123958_38.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a9ee0ca1-ca0a-431d-b3ac-0947c308f5ed_characterizeImage_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a9ee0ca1-ca0a-431d-b3ac-0947c308f5ed_characterizeImage_1123958_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:03 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:03 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:03 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:03 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:03 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:04 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:04 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:04 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:04 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 532 try 1 failed parsl.dataflow.dflow ERROR: Task 532 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 45d6a8ba54f5e6ee2c28eb1095b94851 with result from task 532 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 533 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 533 try 0 failed parsl.dataflow.dflow INFO: Task 533 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7a7dbf51-1b86-4a9e-b4dd-363a98d0cd5f_calibrate_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7a7dbf51-1b86-4a9e-b4dd-363a98d0cd5f_calibrate_1124087_61.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5b402079-d24d-41f1-9bca-6b461d642a8a_characterizeImage_1124087_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:04 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:04 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:05 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 628 try 1 failed parsl.dataflow.dflow ERROR: Task 628 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 196d136254572d1065715edbf8821ef2 with result from task 628 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 629 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 629 try 0 failed parsl.dataflow.dflow INFO: Task 629 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/519f596c-ca4a-463a-9d78-4e3fe3bba319_calibrate_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/519f596c-ca4a-463a-9d78-4e3fe3bba319_calibrate_1124087_59.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ca01823f-dc5d-4ffa-a616-9ce1ba78a13b_characterizeImage_1124087_59.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 577 try 1 failed parsl.dataflow.dflow ERROR: Task 577 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f0f399c40fcb3910e2bafb7f71b9e285 with result from task 577 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 578 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 578 try 0 failed parsl.dataflow.dflow INFO: Task 578 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b4c6a904-2235-4683-ac76-1d4c1741218c_calibrate_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/b4c6a904-2235-4683-ac76-1d4c1741218c_calibrate_1124087_10.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/4c38a7af-3a7b-4d17-afc4-61644d783f3c_characterizeImage_1124087_10.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2a5062f15bd579f4425f01dfce928b2 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 97e5b46aff83aaaedf310307f5ef95df parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca549a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/33a51cf9-50eb-44ad-a7d2-ff7dc961b28e_calibrate_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/33a51cf9-50eb-44ad-a7d2-ff7dc961b28e_calibrate_1123954_61.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/037568f8-0932-4467-bd82-93e364608cba_characterizeImage_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/037568f8-0932-4467-bd82-93e364608cba_characterizeImage_1123954_61.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c16f607ceeaaebb98cd097ddaa67524 with result from task 601 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:05 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 4e4f59454f04af86caa76c427c677505 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e5a78349-5eb3-44dc-be53-0846c589d2bd_calibrate_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e5a78349-5eb3-44dc-be53-0846c589d2bd_calibrate_1123956_28.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/fdd6a21e-23c3-47f8-9b4a-cf3dea418dda_characterizeImage_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/fdd6a21e-23c3-47f8-9b4a-cf3dea418dda_characterizeImage_1123956_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:05 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:05 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d2dc5b4d095d667e43a5912ba971723 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 3ad60b8cff0131934784d401b45cec61 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887324540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/140e54eb-6918-4c25-84dd-217473588fcf_calibrate_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/140e54eb-6918-4c25-84dd-217473588fcf_calibrate_1123954_25.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fed4baa6-4664-4291-8768-e2237c9682f1_characterizeImage_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/fed4baa6-4664-4291-8768-e2237c9682f1_characterizeImage_1123954_25.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aab8db812c239accb75671a14ce495f with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 304e0f467c493f87fe5369687c0fa184 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9683e906-03cd-4639-ad2c-d1503abf2b21_calibrate_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9683e906-03cd-4639-ad2c-d1503abf2b21_calibrate_1123958_46.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c35bca51-eec9-40de-95c4-a229c0f56a5f_characterizeImage_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/c35bca51-eec9-40de-95c4-a229c0f56a5f_characterizeImage_1123958_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:06 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:06 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 625 try 1 failed parsl.dataflow.dflow ERROR: Task 625 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ecfd50c7ba4321186e68d885e4391cd3 with result from task 625 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 626 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 626 try 0 failed parsl.dataflow.dflow INFO: Task 626 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/674c5d02-9867-416f-92aa-c713d60d0269_calibrate_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/674c5d02-9867-416f-92aa-c713d60d0269_calibrate_1124087_31.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/58fb9d76-4624-40f1-925d-e5e9acdee56b_characterizeImage_1124087_31.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75fc9247c1a6d01a7f76216ef5dae977 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 56b9a7a1c16c6f7be37581809e1a8605 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a331070f-50d7-4407-b2ae-d8c8e37e932f_calibrate_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a331070f-50d7-4407-b2ae-d8c8e37e932f_calibrate_1123958_18.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7626e02d-eb24-48bf-b2e3-546c89f2f0ec_characterizeImage_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7626e02d-eb24-48bf-b2e3-546c89f2f0ec_characterizeImage_1123958_18.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6042d6c1d3fbf4e7bb5f8287469354c with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 032a7b14855fcb5fc7077f3f3f2f5721 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f4ab0173-657a-4ac1-9b38-8f21300ada2a_calibrate_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f4ab0173-657a-4ac1-9b38-8f21300ada2a_calibrate_1123956_22.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1b65d045-50b0-4efe-bc2b-6ffef52f5b6a_characterizeImage_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1b65d045-50b0-4efe-bc2b-6ffef52f5b6a_characterizeImage_1123956_22.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 459a13d380f12fdc88ad545b5459f70f with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 034e09c73a339df737155bf5e8c0c398 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/52cb2a01-e283-443a-beae-a275c4b2d842_calibrate_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/52cb2a01-e283-443a-beae-a275c4b2d842_calibrate_1123954_24.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/be5fa37a-5e9a-466f-b147-22da03eb56ab_characterizeImage_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/be5fa37a-5e9a-466f-b147-22da03eb56ab_characterizeImage_1123954_24.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 640 try 1 failed parsl.dataflow.dflow ERROR: Task 640 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0755a13a0b3a9c1581f38e73d6222198 with result from task 640 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 641 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 641 try 0 failed parsl.dataflow.dflow INFO: Task 641 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e9550d39-c460-4d53-b682-038584cfd913_calibrate_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/e9550d39-c460-4d53-b682-038584cfd913_calibrate_1124087_9.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/14807569-8c35-4263-96f1-c05b71f985d2_characterizeImage_1124087_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:08 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:08 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:08 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:08 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc29694e665941ad5a06c6a54239df9 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 9872d740224ee9ad9fed74d8306135c7 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/c0f52e32-3b23-4af5-84fc-f8806536ac81_calibrate_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/c0f52e32-3b23-4af5-84fc-f8806536ac81_calibrate_1123956_56.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47a23f93-9f45-47b8-af63-2c1cbd78f7af_characterizeImage_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47a23f93-9f45-47b8-af63-2c1cbd78f7af_characterizeImage_1123956_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:09 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:09 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56828a78fd22e660a2e5abe48b10d708 with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 8084c58517ad7a74e2a5cbc99ac8a3f9 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887326ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8eaf7ced-2eb1-4f5a-a359-6c51f0ba01f1_calibrate_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8eaf7ced-2eb1-4f5a-a359-6c51f0ba01f1_calibrate_1123958_55.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/12dce86b-1ccc-4bee-99ba-c6c59a6938e4_characterizeImage_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/12dce86b-1ccc-4bee-99ba-c6c59a6938e4_characterizeImage_1123958_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:09 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:10 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:10 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:10 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 682 try 1 failed parsl.dataflow.dflow ERROR: Task 682 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a203cd63add2c8a45d6d51685d1283a9 with result from task 682 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 683 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 683 try 0 failed parsl.dataflow.dflow INFO: Task 683 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7a86602e-ed3d-4018-b5e2-2effa3255bfe_calibrate_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7a86602e-ed3d-4018-b5e2-2effa3255bfe_calibrate_1124087_26.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/c3cc0514-8ecf-412d-af76-cc9aa4fa14ab_characterizeImage_1124087_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 562 try 1 failed parsl.dataflow.dflow ERROR: Task 562 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6648a3efe0e031fc24544e7786ef65cc with result from task 562 parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:11 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 563 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 563 try 0 failed parsl.dataflow.dflow INFO: Task 563 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/c791f13e-4b46-4a1c-99d8-d8faf43c0b5e_calibrate_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/c791f13e-4b46-4a1c-99d8-d8faf43c0b5e_calibrate_1124087_15.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b87b45e9-1610-401c-b046-2fa25ef543f6_characterizeImage_1124087_15.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 379 try 1 failed parsl.dataflow.dflow ERROR: Task 379 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5f048ec3f471c2e4f50e5dcb84b29045 with result from task 379 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 380 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 380 try 0 failed parsl.dataflow.dflow INFO: Task 380 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/518c32f1-269d-4e2a-9e54-98c8be18e406_calibrate_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/518c32f1-269d-4e2a-9e54-98c8be18e406_calibrate_1124087_46.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/42ba928f-a99f-43fb-b4bb-a6e10f18cb1d_characterizeImage_1124087_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:11 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:12 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eae5378f086b2cd1be9bd9c709ee6bd8 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash dd575a9d3e3038c4a1e459590704262f parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/11055724-d5e0-4de2-99a1-4cb4832afca6_calibrate_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/11055724-d5e0-4de2-99a1-4cb4832afca6_calibrate_1123954_53.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/5f7d0e68-f6c5-46b8-b1cb-f6e8f6d8358a_characterizeImage_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/5f7d0e68-f6c5-46b8-b1cb-f6e8f6d8358a_characterizeImage_1123954_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:12 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 193 parsl.dataflow.strategy DEBUG: Executor multi has 193 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 193 parsl.dataflow.strategy DEBUG: Executor multi has 193 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ccc7b5522e1b51791c894da8e1da8b3 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/949fce41-e6c5-4950-8c2c-b8623073126e_calibrate_1123956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/949fce41-e6c5-4950-8c2c-b8623073126e_calibrate_1123956_40.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ab5d35a06af08e900ea87e9e803aeec with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 40c612fa8e51b134891ec13fee4b91f1 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d573f08e-9af0-4b9e-9c89-aa83d0540727_calibrate_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d573f08e-9af0-4b9e-9c89-aa83d0540727_calibrate_1123954_54.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/87bca6a0-35fa-4e7e-b483-f42a45f8fb1b_characterizeImage_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/87bca6a0-35fa-4e7e-b483-f42a45f8fb1b_characterizeImage_1123954_54.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61bd79799455d3c1cab6fe4a54461617 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/705897f1-7735-48bd-a5f2-628649a7f2b9_calibrate_1123958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/705897f1-7735-48bd-a5f2-628649a7f2b9_calibrate_1123958_57.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75a6a849d1a9a01ace0d02bf35976a62 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1ebf18f8-9e7c-45f5-86a9-408224088a5a_calibrate_1123956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1ebf18f8-9e7c-45f5-86a9-408224088a5a_calibrate_1123956_57.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aab8781860c1a80161be6faf15d8e10 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 78351364b504afffb03cbac2f52a5a2f parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/51c50c6a-e675-4a2c-964d-ee9876fb0eff_calibrate_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/51c50c6a-e675-4a2c-964d-ee9876fb0eff_calibrate_1123954_14.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/96b34a2b-abe9-46e1-8c59-ecca997b65fc_characterizeImage_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/96b34a2b-abe9-46e1-8c59-ecca997b65fc_characterizeImage_1123954_14.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f0c908f9b89c7e4d1e9d4de55a3fa74 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9a064421-8484-419c-8b7b-e831b34f8aed_calibrate_1123958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9a064421-8484-419c-8b7b-e831b34f8aed_calibrate_1123958_5.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae559617b2d04171503394cf4965618c with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0a7e2fa2-0580-4d69-bfce-f072ee24443e_calibrate_1123954_41.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0a7e2fa2-0580-4d69-bfce-f072ee24443e_calibrate_1123954_41.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b992e1b3b99ecccf3516d07e09eb645 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/11064b60-d210-4222-92bc-a35d009e23f8_calibrate_1123958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/11064b60-d210-4222-92bc-a35d009e23f8_calibrate_1123958_49.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b53c51e9ec9096b3e8ea3932dde314f9 with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash a9574578c7fa7da0fc6a229a02663ed2 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e8c0d3ed-5d94-4c9c-b13e-b44935477b44_calibrate_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e8c0d3ed-5d94-4c9c-b13e-b44935477b44_calibrate_1123954_37.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3623f8cf-3826-45ed-99d3-27587c1eae06_characterizeImage_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/3623f8cf-3826-45ed-99d3-27587c1eae06_characterizeImage_1123954_37.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5002294924373407b894ccceeafa4111 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 78d2bad458a54521e86137a027f978a5 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9467e45b-b759-49ad-99f0-e78b531eabca_characterizeImage_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9467e45b-b759-49ad-99f0-e78b531eabca_characterizeImage_1123954_31.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 267d7096696584169f2027f5508cfb49 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash e4a7cce9fdce4f7738b4cfb451094690 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468872198a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d81459be-ce21-4efe-ac93-a26c00619f12_calibrate_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d81459be-ce21-4efe-ac93-a26c00619f12_calibrate_1123954_32.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a778b1e4-731f-4273-a848-7c61d5061912_characterizeImage_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/a778b1e4-731f-4273-a848-7c61d5061912_characterizeImage_1123954_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 541 try 1 failed parsl.dataflow.dflow ERROR: Task 541 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a7007ec0a6812f3aab3655f69b526cb6 with result from task 541 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 542 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 542 try 0 failed parsl.dataflow.dflow INFO: Task 542 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/6fdb9679-6725-405f-8480-13fd9d571537_calibrate_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/6fdb9679-6725-405f-8480-13fd9d571537_calibrate_1124087_62.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/5e790302-5d04-4575-83d8-8813514f002e_characterizeImage_1124087_62.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fef4da965a2c2c070ce498aa9f18e652 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash f806c1d56a2b1255a0be6f2d760c962c parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d2ac9435-caa1-4f9f-bf84-3ce8825f094d_calibrate_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d2ac9435-caa1-4f9f-bf84-3ce8825f094d_calibrate_1123956_60.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/cf7d6529-c6d7-43b9-b66d-f6fe104b22ca_characterizeImage_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/cf7d6529-c6d7-43b9-b66d-f6fe104b22ca_characterizeImage_1123956_60.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 933398653b14d0c5e2c9aaf564f2af72 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a82d629-603d-44b6-b8ab-b725bd85eb48_calibrate_1123956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a82d629-603d-44b6-b8ab-b725bd85eb48_calibrate_1123956_10.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e64ff3c30c2c91e344ac17165031b7f with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 3ca8a569772bc88308998cfe41433082 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a47e5e71-1600-499d-b7a5-819fd2299482_calibrate_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a47e5e71-1600-499d-b7a5-819fd2299482_calibrate_1123958_19.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cca48b58-5998-4ffa-9c58-1471409a22f2_characterizeImage_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/cca48b58-5998-4ffa-9c58-1471409a22f2_characterizeImage_1123958_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 388 try 1 failed parsl.dataflow.dflow ERROR: Task 388 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f114c20ac0b64ae0ef146a01e003f477 with result from task 388 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 389 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 389 try 0 failed parsl.dataflow.dflow INFO: Task 389 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7dbfa3f0-d927-420a-8c9b-1482701ee4a4_calibrate_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/7dbfa3f0-d927-420a-8c9b-1482701ee4a4_calibrate_1124087_1.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/ce491cdc-65aa-46db-805a-294485b9a04c_characterizeImage_1124087_1.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 733 try 1 failed parsl.dataflow.dflow ERROR: Task 733 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a6b612697bef76f74dadc87a271f93a4 with result from task 733 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 734 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 734 try 0 failed parsl.dataflow.dflow INFO: Task 734 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/381035f5-60d5-47fd-a4b0-3eb9496e4cab_calibrate_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/381035f5-60d5-47fd-a4b0-3eb9496e4cab_calibrate_1124087_34.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/b1713d61-075e-4932-b626-a103a5e39af5_characterizeImage_1124087_34.stderr 2024-03-07 03:42:22 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:42:22 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:23 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 853b807e8f7d199ea3149ca1f3466932 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc61d3b-008a-40a5-bed9-67f2179c72da_calibrate_1123958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc61d3b-008a-40a5-bed9-67f2179c72da_calibrate_1123958_23.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e51542146c482f2ef1a3d023ac199e5b with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/815181a2-9292-4bbe-8b60-f754670d53b5_calibrate_1123954_50.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/815181a2-9292-4bbe-8b60-f754670d53b5_calibrate_1123954_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:23 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6197411640117f0ff3492f7c2ab81c5 with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 2122e2852cdf4eb68390b2bd752907e5 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2708764c-88dd-4621-89ee-392b83681923_calibrate_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2708764c-88dd-4621-89ee-392b83681923_calibrate_1123956_6.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1c3d480e-ee30-4d31-a94c-85af7d845958_characterizeImage_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/1c3d480e-ee30-4d31-a94c-85af7d845958_characterizeImage_1123956_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:24 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70e57752a0f37b0d9f66b02ac2a3d75d with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/83b3b14f-a546-48c9-936e-aee574d8bd1f_calibrate_1123956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/83b3b14f-a546-48c9-936e-aee574d8bd1f_calibrate_1123956_17.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 655 try 1 failed parsl.dataflow.dflow ERROR: Task 655 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 719af86669e2b5ad28901fca11c249ec with result from task 655 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 656 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 656 try 0 failed parsl.dataflow.dflow INFO: Task 656 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/9d3db634-7336-4e28-8dfe-93a46c399b23_calibrate_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/9d3db634-7336-4e28-8dfe-93a46c399b23_calibrate_1124087_22.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/e1fd000d-07ed-4659-9f2b-88b715cb0183_characterizeImage_1124087_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:24 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e15668ced3c74de525b2db8c9c6d4e2 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash d3385e090b7b46f2bfa97d5bdeb2af82 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ca075393-1115-43c0-8b71-5115b4309c19_characterizeImage_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ca075393-1115-43c0-8b71-5115b4309c19_characterizeImage_1123956_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:24 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb06a8c4a15c6e489e8c68b32bb15f6c with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1b1235bd-c631-4604-a43a-11337a65efaa_calibrate_1123956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/1b1235bd-c631-4604-a43a-11337a65efaa_calibrate_1123956_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:25 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f469e8774b3789501d41763b9a7edae3 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash bbf7ee67759e87ac8819f99a82192a2c parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/7f320911-e7e1-4e13-9bf4-dc1dc57a8fd9_calibrate_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/7f320911-e7e1-4e13-9bf4-dc1dc57a8fd9_calibrate_1123958_60.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8e867797-5eda-4eba-b69a-39258ff4d317_characterizeImage_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/8e867797-5eda-4eba-b69a-39258ff4d317_characterizeImage_1123958_60.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ff7ac33814f13ed4072e0169e215eb0 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5703dc4-03da-480d-b13b-26d6932fe836_calibrate_1123954_40.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5703dc4-03da-480d-b13b-26d6932fe836_calibrate_1123954_40.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e513f4c7c1d86843c654d44bdb47291 with result from task 229 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:25 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash e6c341e477a8239085ff1593f8b2048d parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/38ff1286-48b9-42b9-916c-202e7c3a1305_characterizeImage_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/38ff1286-48b9-42b9-916c-202e7c3a1305_characterizeImage_1123958_61.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4daa399be1c1b19f2febb1e92e67af3 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/366b0ea8-24be-436c-97fc-d37a7ed5ce2e_calibrate_1123956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/366b0ea8-24be-436c-97fc-d37a7ed5ce2e_calibrate_1123956_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:25 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f049cc302af2bb2e1f32f74aa777c7e with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/85aeebe7-9570-4398-a155-7931368f6a4d_calibrate_1123954_47.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/85aeebe7-9570-4398-a155-7931368f6a4d_calibrate_1123954_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:25 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:26 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 511b6d8030886d24418b85fdbadb73cd with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/681d9e66-37fc-41e0-9ef7-c32d06220f5a_calibrate_1123956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/681d9e66-37fc-41e0-9ef7-c32d06220f5a_calibrate_1123956_4.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 756e6f7ff78a8cf4e5ff4a1668a2911c with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 3786280ec3208ff8d21f1e80a691efe9 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c171580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bffb2ee3-04c9-4aab-bd16-c9911cd4f79d_calibrate_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bffb2ee3-04c9-4aab-bd16-c9911cd4f79d_calibrate_1123956_59.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83834772-88d8-4d01-82af-53beb33c78b8_characterizeImage_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/83834772-88d8-4d01-82af-53beb33c78b8_characterizeImage_1123956_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:26 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:27 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:27 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:27 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:28 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:28 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce7f561deed000581c63402769d0a8d5 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/021dfc23-99cb-400a-b92d-7e7c8db5129d_calibrate_1123954_42.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/021dfc23-99cb-400a-b92d-7e7c8db5129d_calibrate_1123954_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:28 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:29 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:29 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:29 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:30 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:30 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 742 try 1 failed parsl.dataflow.dflow ERROR: Task 742 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry efa1be6b475a8acc5557333513890141 with result from task 742 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 743 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 743 try 0 failed parsl.dataflow.dflow INFO: Task 743 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/12944226-be54-4316-9ca9-492e53628ee4_calibrate_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/12944226-be54-4316-9ca9-492e53628ee4_calibrate_1124087_53.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/8af0a413-6a2a-4226-9fec-e3799a6e509b_characterizeImage_1124087_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:30 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:31 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 367 try 1 failed parsl.dataflow.dflow ERROR: Task 367 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5b350e802da89d3a6ed368f12907824b with result from task 367 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 368 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 368 try 0 failed parsl.dataflow.dflow INFO: Task 368 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/70fac819-b3e9-4cff-9f9d-5b3814d6db90_calibrate_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/70fac819-b3e9-4cff-9f9d-5b3814d6db90_calibrate_1124087_41.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/1572a898-6b59-4b8f-8e17-2e180a654c85_characterizeImage_1124087_41.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab7b8a5cbc5f7ae44eb443e69ed9e36d with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f34c647f-4d7c-4af8-b28c-6ad0ceb45d65_calibrate_1123954_30.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f34c647f-4d7c-4af8-b28c-6ad0ceb45d65_calibrate_1123954_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:31 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:31 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aea96f0019e8ff0167c8db43d6329aca with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 1273719bd752652c2054fff9fc44be3a parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d05cd663-e050-4a4a-a9b4-765c7387e974_calibrate_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d05cd663-e050-4a4a-a9b4-765c7387e974_calibrate_1123958_29.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3393ebd2-9787-4b7f-81c7-6aa5b7e56868_characterizeImage_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3393ebd2-9787-4b7f-81c7-6aa5b7e56868_characterizeImage_1123958_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:32 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 358 try 1 failed parsl.dataflow.dflow ERROR: Task 358 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6285ea3f51196c4bee54f7e2ecf04112 with result from task 358 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 359 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 359 try 0 failed parsl.dataflow.dflow INFO: Task 359 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/0ddfc611-73dc-496e-9b57-729fb8e474e1_calibrate_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/0ddfc611-73dc-496e-9b57-729fb8e474e1_calibrate_1124087_35.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/29a5aeba-9ae8-4e79-ba2f-8ea4783acc63_characterizeImage_1124087_35.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 172 try 1 failed parsl.dataflow.dflow ERROR: Task 172 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6834881ba9fc070da018d95407d6c1d9 with result from task 172 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 173 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 173 try 0 failed parsl.dataflow.dflow INFO: Task 173 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/a142858b-8c8b-4c95-a98d-638b316529cf_calibrate_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/a142858b-8c8b-4c95-a98d-638b316529cf_calibrate_1124087_42.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/65fa5b6a-d07f-4a8e-994d-bb9ad5a00b2b_characterizeImage_1124087_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:32 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:32 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 745018cc608a6b2cf4a5e13d9bb4f365 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b8704ac9-76bd-4585-9e59-fab8b644a334_calibrate_1123956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b8704ac9-76bd-4585-9e59-fab8b644a334_calibrate_1123956_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:33 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 184713d2b907cfd9978eb0312b39d834 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d3d38078-588e-47eb-848e-096c4b6b11b4_calibrate_1123958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d3d38078-588e-47eb-848e-096c4b6b11b4_calibrate_1123958_45.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 075ecd3dab71eeb1bba1d8e34931eee5 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/cb6bed30-baf7-4328-b420-e5aeecc30358_calibrate_1123954_43.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/cb6bed30-baf7-4328-b420-e5aeecc30358_calibrate_1123954_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:33 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:33 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 819bb9dc9535673193f8c398945b5820 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/920a4d47-0957-49d3-9984-76565e4d1bb4_calibrate_1123954_16.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/920a4d47-0957-49d3-9984-76565e4d1bb4_calibrate_1123954_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:34 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a5089ac34d921827cdeb741f7045c5c with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 8bb5bc823e040ff7f0f64a24397fba38 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:34 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc0affe-281f-4e7d-89d1-165c36d8b9f6_calibrate_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc0affe-281f-4e7d-89d1-165c36d8b9f6_calibrate_1123958_8.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/da703fee-55ad-4028-ad47-fc2b316033ee_characterizeImage_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/da703fee-55ad-4028-ad47-fc2b316033ee_characterizeImage_1123958_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 373 try 1 failed parsl.dataflow.dflow ERROR: Task 373 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a7bb50e5d8074ea951d1cf4dc9701aa5 with result from task 373 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 374 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 374 try 0 failed parsl.dataflow.dflow INFO: Task 374 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/871ed5c5-6ad9-41dd-a8b6-41c065d3beeb_calibrate_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/871ed5c5-6ad9-41dd-a8b6-41c065d3beeb_calibrate_1124087_52.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/126f6f1f-f059-436e-8706-b4c697397e25_characterizeImage_1124087_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:34 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77364d45ebafc6d3d4b742ea57ea66bc with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/0aecadd2-fc62-4d7a-9adc-fae213fe9da7_calibrate_1123956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/0aecadd2-fc62-4d7a-9adc-fae213fe9da7_calibrate_1123956_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:35 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:35 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:35 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:36 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33f70c59387f5611cdd73c937fd33c5b with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9801a45f-d654-4b71-a1d0-333e20cd08f2_calibrate_1123956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9801a45f-d654-4b71-a1d0-333e20cd08f2_calibrate_1123956_45.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8207ae6ecd63d1d51e1bd962df9a8d with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash ed32013fce11d75c1336dd528a7f3453 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/04d7eb24-c31c-42ce-8ac7-cafdfaf956dc_calibrate_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/04d7eb24-c31c-42ce-8ac7-cafdfaf956dc_calibrate_1123954_56.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f3660c5c-b3c7-41ab-a673-60bf516a53c1_characterizeImage_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/f3660c5c-b3c7-41ab-a673-60bf516a53c1_characterizeImage_1123954_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:36 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:36 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93ad45a02fed0c89e67d58c411dc7355 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4af15a9b-aa94-484d-8156-3de1ad9e33cf_calibrate_1123958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4af15a9b-aa94-484d-8156-3de1ad9e33cf_calibrate_1123958_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:37 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:37 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:37 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f224d101170a1919d232bda50881504 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6eed9cc0-dc76-498b-a4fb-13f09d4f5cf9_calibrate_1123958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6eed9cc0-dc76-498b-a4fb-13f09d4f5cf9_calibrate_1123958_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 364 try 1 failed parsl.dataflow.dflow ERROR: Task 364 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 971fa46d26c1ddde2f6c5eab813ba3e4 with result from task 364 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 365 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 365 try 0 failed parsl.dataflow.dflow INFO: Task 365 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/6da49ed5-b469-4373-bad9-41fb38d156bf_calibrate_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1124087/6da49ed5-b469-4373-bad9-41fb38d156bf_calibrate_1124087_38.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1124087/fdb7f7db-5f43-4256-a9a7-53733b945b02_characterizeImage_1124087_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:38 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 816b0894c856ee7774b435e614e016d4 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 9be790545c6c7ca45483c5d63d27308f parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8ac84be-4229-435b-bd6c-1f47992b2294_calibrate_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8ac84be-4229-435b-bd6c-1f47992b2294_calibrate_1123954_59.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1cdc9e05-ab4c-4e96-88b4-702eb1b7beac_characterizeImage_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/1cdc9e05-ab4c-4e96-88b4-702eb1b7beac_characterizeImage_1123954_59.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b63695bd2181a0b26b1200fba664e0c5 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/47ae67b5-d348-442d-a349-9b4ef8ba95ba_calibrate_1123954_49.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/47ae67b5-d348-442d-a349-9b4ef8ba95ba_calibrate_1123954_49.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68611d73e989f39100db4020050d69e6 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8ed23f33-0bfc-4126-b792-cc72ded12cd7_calibrate_1123958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8ed23f33-0bfc-4126-b792-cc72ded12cd7_calibrate_1123958_3.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2b5e0a755002e64a75d3676a55a9099 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b705e1ad-7c88-4037-a36e-ff78913a8ca0_calibrate_1123958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b705e1ad-7c88-4037-a36e-ff78913a8ca0_calibrate_1123958_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:40 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:40 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:40 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5100260503e5ee9aa42866289d90368d with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/92bcf368-ea65-4c9b-b0b1-14eaa81253c9_calibrate_1123958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/92bcf368-ea65-4c9b-b0b1-14eaa81253c9_calibrate_1123958_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:41 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84d914368bb4a738d2615e2756d3647a with result from task 137 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:41 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e9f9278d-0f84-4a99-872c-acf3f465a52b_calibrate_1123954_17.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e9f9278d-0f84-4a99-872c-acf3f465a52b_calibrate_1123954_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:41 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:42 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:42 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b59827f959eb5948690984e1e5e8938c with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2512d812-1576-41a2-975a-b50de224c617_calibrate_1123958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2512d812-1576-41a2-975a-b50de224c617_calibrate_1123958_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:42 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:43 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:43 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:43 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:44 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:44 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8180c94b18f675755141d64ec52d82a4 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 5e547113a9d3722b6c3d3423e61789b6 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688737cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8dcce1d1-e20a-4f0f-8049-909e532f014e_calibrate_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8dcce1d1-e20a-4f0f-8049-909e532f014e_calibrate_1123956_53.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/348d183f-0c66-4e3e-8b48-09aa22d1cc23_characterizeImage_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/348d183f-0c66-4e3e-8b48-09aa22d1cc23_characterizeImage_1123956_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:45 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 152 parsl.dataflow.strategy DEBUG: Executor multi has 152 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 152 parsl.dataflow.strategy DEBUG: Executor multi has 152 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3785408078ddf8da4a41c61d57e0b92 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e8c128fb-48d3-4f38-bf18-70861ad5d539_calibrate_1123956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e8c128fb-48d3-4f38-bf18-70861ad5d539_calibrate_1123956_25.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3dc5697ee8f4a4bfbbdfb2ce1060ea3 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 035325d34b6f9ed82f903cdb86c40656 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c12a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/dee842f0-5517-43d3-9825-35db4d902284_calibrate_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/dee842f0-5517-43d3-9825-35db4d902284_calibrate_1123956_8.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae3c5d37-483c-47c4-a495-79f0afae9d04_characterizeImage_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/ae3c5d37-483c-47c4-a495-79f0afae9d04_characterizeImage_1123956_8.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 742b88f7bcf26c8a6084ac60d5ea8940 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/925bacec-f40a-42e8-a537-b04b020d4b6d_calibrate_1123958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/925bacec-f40a-42e8-a537-b04b020d4b6d_calibrate_1123958_12.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73be789cfea528f945251e816546ff4 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/ffccb6f1-e967-4c2e-b759-fbbda93173f6_calibrate_1123958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/ffccb6f1-e967-4c2e-b759-fbbda93173f6_calibrate_1123958_28.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b73c3a899ba5db2dc0d09cba6e484212 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/94a79782-0355-43c1-b875-5bf6efae70f5_calibrate_1123956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/94a79782-0355-43c1-b875-5bf6efae70f5_calibrate_1123956_32.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df670a7a974ff073f3df864693f0a822 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash a0773a97ba6500c405d17f2235b08777 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f848cccf-5440-4bf9-a9b3-0e2427d86988_calibrate_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f848cccf-5440-4bf9-a9b3-0e2427d86988_calibrate_1123958_6.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3bfa88a7-8dac-44c1-830e-9634a126d696_characterizeImage_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/3bfa88a7-8dac-44c1-830e-9634a126d696_characterizeImage_1123958_6.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379feab4d8f2b8b12a258a54a011cecd with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/252b7a2d-b964-4523-aec9-1b469efbc51c_calibrate_1123958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/252b7a2d-b964-4523-aec9-1b469efbc51c_calibrate_1123958_9.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0b672ccd51d236aa154502bb838c039 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c696acbd-d97f-4607-8c28-5ae5e5f5babe_calibrate_1123954_48.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c696acbd-d97f-4607-8c28-5ae5e5f5babe_calibrate_1123954_48.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 058fa20089f0a709d59277db0396c23f with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/38816df2-756b-4533-acc2-8f84ddd6850a_calibrate_1123956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/38816df2-756b-4533-acc2-8f84ddd6850a_calibrate_1123956_41.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0af3d35ed3e45ae837a6d3959e758c2e with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/76f4bfb0-764d-403b-b11f-ddc4ed8f8ac5_calibrate_1123958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/76f4bfb0-764d-403b-b11f-ddc4ed8f8ac5_calibrate_1123958_22.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f56b19465c80674c0e3edb51525d4de with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1de7179-439a-428d-8fbe-21d927cfc40e_calibrate_1123954_23.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1de7179-439a-428d-8fbe-21d927cfc40e_calibrate_1123954_23.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 143 parsl.dataflow.strategy DEBUG: Executor multi has 143 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9491652ccf51282d8a88ded5450e2ea1 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e38973ce-250c-4b56-a7ee-fada127fc56e_calibrate_1123954_46.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e38973ce-250c-4b56-a7ee-fada127fc56e_calibrate_1123954_46.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9359b08b4cbc13e9df4a3fa8835bf588 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/dc9099b5-c418-479e-859e-be93f55a3e02_calibrate_1123958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/dc9099b5-c418-479e-859e-be93f55a3e02_calibrate_1123958_36.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de585e852ac156090caf58717ebd90ff with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/518ba6cd-2900-44b0-9b85-0bda526de457_calibrate_1123958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/518ba6cd-2900-44b0-9b85-0bda526de457_calibrate_1123958_51.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3068f8d35c5760ed81b9f57037fc3e57 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash ab05cc486844205fe02961cc37ddd06a parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688721a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/73762187-013f-4e6a-8050-96671123684e_calibrate_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/73762187-013f-4e6a-8050-96671123684e_calibrate_1123958_34.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a0c07868-8341-4c04-b2b2-94da15452a2f_characterizeImage_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/a0c07868-8341-4c04-b2b2-94da15452a2f_characterizeImage_1123958_34.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3ee498c5a4bdb965116b986cb6c00e6 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/4c606afb-fb32-4a38-811e-e26fac9500b3_calibrate_1123954_3.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/4c606afb-fb32-4a38-811e-e26fac9500b3_calibrate_1123954_3.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82cff02b0ac0d441156a5852d2aa592d with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/13aa2b05-9cb9-49fe-961a-893fea03bdd8_calibrate_1123954_36.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/13aa2b05-9cb9-49fe-961a-893fea03bdd8_calibrate_1123954_36.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02faf2d0109f697d03a513da10f20cb0 with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 5a6ffd3f2b32c952e11f56fb492720c1 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/26a30aad-1ebc-4af5-ae8c-58c43707557d_calibrate_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/26a30aad-1ebc-4af5-ae8c-58c43707557d_calibrate_1123958_59.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/675b26ec-72f7-45e6-b143-f6e9a216ced7_characterizeImage_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/675b26ec-72f7-45e6-b143-f6e9a216ced7_characterizeImage_1123958_59.stderr 2024-03-07 03:42:55 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:42:55 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:55 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:56 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:56 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:57 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a648495086374689b71a01a074acb8d2 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1b24a157-83d9-4db7-a323-46a8c8e332b2_calibrate_1123958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1b24a157-83d9-4db7-a323-46a8c8e332b2_calibrate_1123958_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:57 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c93abf94c3e754b8e427cc8db310546 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8747c728-9618-4175-a9b2-bd478be58843_calibrate_1123958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8747c728-9618-4175-a9b2-bd478be58843_calibrate_1123958_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:57 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3373ec63ae88d27bd298db3eea7dd1d with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 62479e0ec446c0319b62e6fab33b1563 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468872185e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b73fcedb-51f9-42c9-9c05-e07c0cc8a606_calibrate_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b73fcedb-51f9-42c9-9c05-e07c0cc8a606_calibrate_1123956_13.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e1f8447-0f54-448b-8e32-2643b854c7b1_characterizeImage_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/8e1f8447-0f54-448b-8e32-2643b854c7b1_characterizeImage_1123956_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:58 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23105cf2a04303904f0f1a23d72906ad with result from task 361 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:58 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 320e66b948ad10f7f495b00fc6e933ef parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3613447d-4853-4003-abb3-5f0137e0a792_calibrate_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3613447d-4853-4003-abb3-5f0137e0a792_calibrate_1123958_21.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/48e969f6-0671-4108-864b-24691d29e6e5_characterizeImage_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/48e969f6-0671-4108-864b-24691d29e6e5_characterizeImage_1123958_21.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e11826d4112b7981d8cac5092e09cc7 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ec6fdc32-f80d-4c5f-a142-815458209655_calibrate_1123956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ec6fdc32-f80d-4c5f-a142-815458209655_calibrate_1123956_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:58 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:59 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b0ac6a77dffd5a2b75ed1ce0cab8b7 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8936eb21-2564-408f-b10a-006657e0bb7e_calibrate_1123958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8936eb21-2564-408f-b10a-006657e0bb7e_calibrate_1123958_56.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57c8c479cf660299092fcdcb1183b5c4 with result from task 203 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:42:59 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0d22d6ec-7fe3-4210-aede-8c2689b2aecb_calibrate_1123954_19.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0d22d6ec-7fe3-4210-aede-8c2689b2aecb_calibrate_1123954_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:00 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:00 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03779c58a5a7df31c5d5a341f42f58a2 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f1269283-56ab-4c3e-a5e8-4ea9bfe11140_calibrate_1123958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f1269283-56ab-4c3e-a5e8-4ea9bfe11140_calibrate_1123958_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:00 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:01 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:01 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:01 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:02 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:02 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:02 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:03 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:03 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25487ff0e1e2ba59f1a2b98536a39e1a with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/059f69a4-3175-4420-b2df-6204da0c5355_calibrate_1123956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/059f69a4-3175-4420-b2df-6204da0c5355_calibrate_1123956_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:03 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 356e4f5e2a1b5487b84ef08e21a20e90 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/090a2508-cf75-4e94-a345-4acd230ba515_calibrate_1123956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/090a2508-cf75-4e94-a345-4acd230ba515_calibrate_1123956_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:04 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33d752cc18567d5e3340e32b503671cd with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/674c9a65-b0f5-4d10-accf-626a6190a9c2_calibrate_1123954_58.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/674c9a65-b0f5-4d10-accf-626a6190a9c2_calibrate_1123954_58.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37ed6f6ee0a7006ae24f81cde008de55 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e7c0f3ca-e123-4c32-a05d-214dee52a7eb_calibrate_1123956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e7c0f3ca-e123-4c32-a05d-214dee52a7eb_calibrate_1123956_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:04 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:04 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:05 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:05 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41b81a8a366e8b489d6e99215b53924f with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 537461d508ddc3d2b96e055f61359f14 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887219ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ebc6b77a-9823-4ef7-86ba-351aaebb84db_calibrate_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ebc6b77a-9823-4ef7-86ba-351aaebb84db_calibrate_1123956_54.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/09333de0-8ee2-4517-81b8-b34948bceea4_characterizeImage_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/09333de0-8ee2-4517-81b8-b34948bceea4_characterizeImage_1123956_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:06 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:06 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 323 try 1 failed parsl.dataflow.dflow ERROR: Task 323 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 91ad19b921b3396663b2b24ea71d25d6 with result from task 323 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9ba3b54a-7e6c-4a4b-a45c-dde2c913714f_calibrate_1123958_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:06 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:07 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 963c501507c4256937577b725b45a487 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/fc55c613-b0f8-49cc-81ff-55953684f687_calibrate_1123954_18.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/fc55c613-b0f8-49cc-81ff-55953684f687_calibrate_1123954_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:07 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50f1584425a05ba6ecdeff8fb8b0e7a5 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/7156f0d5-7366-459e-a1dc-b198fb475464_calibrate_1123956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/7156f0d5-7366-459e-a1dc-b198fb475464_calibrate_1123956_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:07 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:08 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:08 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:08 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:08 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:09 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:09 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2463773518f9af1b2d9c73fcc0f3179 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b870d33a-390a-4843-a887-4ee657867948_calibrate_1123954_51.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b870d33a-390a-4843-a887-4ee657867948_calibrate_1123954_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:09 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:09 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 529bf28e55e0b4ef4679a7804a13d08b with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0686fa32-3e9f-448a-bbfe-ca5053017ee3_calibrate_1123954_9.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0686fa32-3e9f-448a-bbfe-ca5053017ee3_calibrate_1123954_9.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2476573dfc79c900e8be8449e984fec7 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/164e6172-a1ac-4ab8-976a-971b226cc3a7_calibrate_1123954_29.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/164e6172-a1ac-4ab8-976a-971b226cc3a7_calibrate_1123954_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:09 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:09 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d16416eb486ec9a978a47cbae6c3d92 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/99971dc1-2d01-4f76-a718-a35c8c28dd15_calibrate_1123954_26.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/99971dc1-2d01-4f76-a718-a35c8c28dd15_calibrate_1123954_26.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c147eee541078792d536ac5abb377d69 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6011b5e3-47d1-4fde-ad5f-3988865eb172_calibrate_1123958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6011b5e3-47d1-4fde-ad5f-3988865eb172_calibrate_1123958_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:12 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:12 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:12 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 580168d4b7a60a50e70ba3551982b47a with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/6a4994d0-91b0-47c1-aad6-5fea3818552e_calibrate_1123956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/6a4994d0-91b0-47c1-aad6-5fea3818552e_calibrate_1123956_21.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9757fe30d0d8628a807993669ae281e9 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/7090cd44-f099-4e36-bfa0-2b197a6982b5_calibrate_1123954_1.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/7090cd44-f099-4e36-bfa0-2b197a6982b5_calibrate_1123954_1.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac088d4ab9a7dc529fe30dfbc1d005c3 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e2c58479-97b5-4194-b88c-708b4c277f28_calibrate_1123954_21.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e2c58479-97b5-4194-b88c-708b4c277f28_calibrate_1123954_21.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a0ec5abcbdfc7dc9f08175b8ebf289c with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/21958c61-1506-41f0-bbf1-cff275140151_calibrate_1123956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/21958c61-1506-41f0-bbf1-cff275140151_calibrate_1123956_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:13 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5539020be09867ea80d52e2e8aac8dd with result from task 560 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:13 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4b618195-c53b-4236-aa82-93873f6edeb9_calibrate_1123958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/4b618195-c53b-4236-aa82-93873f6edeb9_calibrate_1123958_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:14 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6cbe053aaafaa58736ee00008b39e8c with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0fdadd55-8fc6-4439-bd66-01b4782525ef_calibrate_1123954_7.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0fdadd55-8fc6-4439-bd66-01b4782525ef_calibrate_1123954_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:14 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:14 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:15 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b746173c5ba4ec785f45c02937adad3 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bf2de8d3-9459-4c41-b299-784297e6072d_calibrate_1123956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bf2de8d3-9459-4c41-b299-784297e6072d_calibrate_1123956_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:15 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:15 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:16 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b4f42b155308b88c47cf9e22d1ff170 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f29c82e1-c6ba-4f5b-814d-94f8921f09fd_calibrate_1123958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f29c82e1-c6ba-4f5b-814d-94f8921f09fd_calibrate_1123958_10.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6496ad6a401ebc93523fe8aa11c4b711 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9f99a4be-be7b-41b4-8b58-0e601e972f18_calibrate_1123956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/9f99a4be-be7b-41b4-8b58-0e601e972f18_calibrate_1123956_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:16 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb491882ba33593e52ca13c5924aa56 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5022d58-4829-4c04-ae86-98d8d391e4b9_calibrate_1123954_13.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f5022d58-4829-4c04-ae86-98d8d391e4b9_calibrate_1123954_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:16 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:17 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2316d1f639034ed1bac079f2d51c10d4 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f6544871-761f-416e-9e94-b04155f34cb8_calibrate_1123954_38.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/f6544871-761f-416e-9e94-b04155f34cb8_calibrate_1123954_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:17 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 109 parsl.dataflow.strategy DEBUG: Executor multi has 109 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 109 parsl.dataflow.strategy DEBUG: Executor multi has 109 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61d7810220a85052058c787ea99513c6 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2b8867a3-cfa2-4092-950e-c519f7eda92d_calibrate_1123956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2b8867a3-cfa2-4092-950e-c519f7eda92d_calibrate_1123956_14.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25b8ceb4f5019a641878f19f8bbaeb59 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash e3dbc2f7ed4c220e42eae8e8d2f15982 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cfc2015c-8db9-4ad9-ae62-bea358caff5c_calibrate_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cfc2015c-8db9-4ad9-ae62-bea358caff5c_calibrate_1123958_25.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/4d882cf5-8ee0-47e8-9071-f285a95db116_characterizeImage_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/4d882cf5-8ee0-47e8-9071-f285a95db116_characterizeImage_1123958_25.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71bf5a94c99d18879b298e3ef0b271a0 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/2bc1ddab-71b1-4119-b213-94c147acb2e8_calibrate_1123954_5.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/2bc1ddab-71b1-4119-b213-94c147acb2e8_calibrate_1123954_5.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b151e053cd03994ff9b25d928a4a50fa with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9f68b607-8728-40e0-9b47-b580076af5b7_calibrate_1123958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9f68b607-8728-40e0-9b47-b580076af5b7_calibrate_1123958_4.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5514599037d040068e8eddd0a2ccba4 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/343aa867-b2e1-4c04-9ce5-ec6e9d80da9a_calibrate_1123958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/343aa867-b2e1-4c04-9ce5-ec6e9d80da9a_calibrate_1123958_26.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aaefafb41630e2d1de875b4b058998a with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/190331a3-0fd9-4a30-8643-a8f832b0bdf0_calibrate_1123958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/190331a3-0fd9-4a30-8643-a8f832b0bdf0_calibrate_1123958_1.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce1042bd3899d55d72c5234cbddd742f with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bba558cd-055e-4ea2-b9ef-1693ea20ef63_calibrate_1123958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bba558cd-055e-4ea2-b9ef-1693ea20ef63_calibrate_1123958_44.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd32498e4d96904e141c4b949bf281de with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/eb92e0e6-59e0-43e5-b892-6d80e468e291_calibrate_1123954_55.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/eb92e0e6-59e0-43e5-b892-6d80e468e291_calibrate_1123954_55.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 102 parsl.dataflow.strategy DEBUG: Executor multi has 102 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07974d7b11152256b90954224d50e988 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/de7ffc46-9b78-4d63-ad15-615909fc1767_calibrate_1123954_52.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/de7ffc46-9b78-4d63-ad15-615909fc1767_calibrate_1123954_52.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5fa69650e8178ed488c48cf98ae8728 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9d289693-d253-44bb-956c-5dfa24d61553_calibrate_1123958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9d289693-d253-44bb-956c-5dfa24d61553_calibrate_1123958_14.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81ebc3d86a9cf2b9e158aca61b1b7429 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/330a729b-5f3d-4254-9844-206afc6b006a_calibrate_1123954_4.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/330a729b-5f3d-4254-9844-206afc6b006a_calibrate_1123954_4.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d3068aca78df540e7220f305945397 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5249c515-ffc9-484c-988a-96e480778854_calibrate_1123958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5249c515-ffc9-484c-988a-96e480778854_calibrate_1123958_24.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94baa9fdf30bdc8c9cfa57937d3a08c2 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e3d8aac8-b3b7-4883-848e-c79b8c82dc59_calibrate_1123954_45.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e3d8aac8-b3b7-4883-848e-c79b8c82dc59_calibrate_1123954_45.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d031d13835f8958a0fb3b15617ff683 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ef550ae2-9ed4-4a4c-9f53-3d13427cd12d_calibrate_1123956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ef550ae2-9ed4-4a4c-9f53-3d13427cd12d_calibrate_1123956_49.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0157cc767ad343edd201bbd7978c5e3 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 729b2eb05b123063b2b85b9c463b7983 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7f5965a2-75f2-4d4b-9ad5-a9fb67b0768f_characterizeImage_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/7f5965a2-75f2-4d4b-9ad5-a9fb67b0768f_characterizeImage_1123958_33.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0902645de332b05718e9eb202b24ccf3 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8d2b3b41-6ba8-4210-a4ef-dd99150c93cd_calibrate_1123958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8d2b3b41-6ba8-4210-a4ef-dd99150c93cd_calibrate_1123958_43.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e710048414e777f9fc0342a5e947df88 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/56e3f47c-b515-456a-8d0e-39487889a2aa_calibrate_1123958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/56e3f47c-b515-456a-8d0e-39487889a2aa_calibrate_1123958_48.stderr 2024-03-07 03:43:27 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:43:27 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 938bd585f197d7529c165427d00c58fe with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d8d86925-a4ce-4cf0-a254-a02f13095036_calibrate_1123956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d8d86925-a4ce-4cf0-a254-a02f13095036_calibrate_1123956_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:28 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:28 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4ecc67abe9852bdff537342ef578d70 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0da6ab02-b884-49ce-a19d-cd402a6658df_calibrate_1123954_57.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/0da6ab02-b884-49ce-a19d-cd402a6658df_calibrate_1123954_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:29 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:29 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 004fae222d149b5d5f2007975fbaba87 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/58c40b57-818d-49a7-8440-bfcae84b7d04_calibrate_1123958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/58c40b57-818d-49a7-8440-bfcae84b7d04_calibrate_1123958_7.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 256bbe1f8521e4141239fdafbb273dea with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d22b666f-da5a-42dd-be7d-f059c1d5c5fb_calibrate_1123954_60.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d22b666f-da5a-42dd-be7d-f059c1d5c5fb_calibrate_1123954_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:29 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08a38ee395e12e786e84f38c54bc649e with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/db3d3865-ea93-4fe3-913a-fcbe9294d1c0_calibrate_1123956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/db3d3865-ea93-4fe3-913a-fcbe9294d1c0_calibrate_1123956_18.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 176 try 0 failed parsl.dataflow.dflow INFO: Task 176 marked for retry parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 78d2bad458a54521e86137a027f978a5 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688caa1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 1 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:30 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:30 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c45314f21d8259322d595e77ca19ea3 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/a9ab9d0c-5061-4760-9f5e-2d379ddd0f6f_calibrate_1123956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/a9ab9d0c-5061-4760-9f5e-2d379ddd0f6f_calibrate_1123956_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:30 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea77285ca7c6387f26fe98651d636b85 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e55a5b10-71db-40ea-9908-03190d54d021_calibrate_1123956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e55a5b10-71db-40ea-9908-03190d54d021_calibrate_1123956_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:31 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8b191a99d71ab0a5de1cc728ecf059 with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6cf0bab4-752b-4689-bc35-d04032699e59_calibrate_1123958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/6cf0bab4-752b-4689-bc35-d04032699e59_calibrate_1123958_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:31 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e40ead427df9d7ae3b8442f5f06d25e with result from task 524 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:31 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8ae4d37c-57d1-481b-a107-4ff840c3d6b1_calibrate_1123954_39.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8ae4d37c-57d1-481b-a107-4ff840c3d6b1_calibrate_1123954_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:32 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:32 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b122bca9b780c7c85115f6bd323bc64a with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 778753578739caab05073b66334dca97 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/186c4a1b-c904-4842-b566-e965de77073a_calibrate_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/186c4a1b-c904-4842-b566-e965de77073a_calibrate_1123956_33.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/546ef7bd-7e28-432c-add2-2cb9bb0b4c4e_characterizeImage_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/546ef7bd-7e28-432c-add2-2cb9bb0b4c4e_characterizeImage_1123956_33.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8df78e5dc09e6f4cef3b07a72799530d with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4309fd8d-c694-4775-8b65-3a585a1fc71e_calibrate_1123956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4309fd8d-c694-4775-8b65-3a585a1fc71e_calibrate_1123956_20.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f84a0f8ac9b6fba5754888552c0285b7 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/0370d87b-8238-484e-9b01-cb0abbde6feb_calibrate_1123958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/0370d87b-8238-484e-9b01-cb0abbde6feb_calibrate_1123958_41.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dd043c7b1d839d4e82f377016d3125d with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b8481e0c-19d9-4438-b890-c8b6bdaa0646_calibrate_1123958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/b8481e0c-19d9-4438-b890-c8b6bdaa0646_calibrate_1123958_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:33 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:33 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af76411577c0c46d8efa546d33016b09 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9e53aea9-168b-476d-92ab-a8fed3fe3aa2_calibrate_1123958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9e53aea9-168b-476d-92ab-a8fed3fe3aa2_calibrate_1123958_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:33 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:34 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:34 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:34 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:35 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:35 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:35 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:36 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d6b2ad12e0ee5d08607a02eb9ed6d4b with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/00356e81-5d7b-4d26-bda4-40cb6862533e_calibrate_1123956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/00356e81-5d7b-4d26-bda4-40cb6862533e_calibrate_1123956_27.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 356 try 0 failed parsl.dataflow.dflow INFO: Task 356 marked for retry parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash d3385e090b7b46f2bfa97d5bdeb2af82 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 1 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:36 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72a77d83b0f53e140422d048d358eeb8 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/20ede348-3df5-414a-82c2-9972dd218fea_calibrate_1123958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/20ede348-3df5-414a-82c2-9972dd218fea_calibrate_1123958_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:36 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2b6e81e1bb84e8e876d346e819cb7f9 with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/81e2d5fa-3c53-4669-af47-045ea748ab13_calibrate_1123956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/81e2d5fa-3c53-4669-af47-045ea748ab13_calibrate_1123956_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:37 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73f85a4a27ff2a45b01a2b5d010a0511 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/39d257ef-e5ce-4a69-8665-2eecd2c5d6df_calibrate_1123956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/39d257ef-e5ce-4a69-8665-2eecd2c5d6df_calibrate_1123956_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:37 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:37 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:38 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:38 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:38 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:39 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5000c0f68db8e2fe6b223c079b40f4e with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/41f9f517-aa10-4afa-8b42-7e22155d1a4c_calibrate_1123954_62.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/41f9f517-aa10-4afa-8b42-7e22155d1a4c_calibrate_1123954_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:39 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:40 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:40 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:40 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0b2884dfcf727b43e565a26211c3325 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 7358c250f7750e4fd9df4967ab532469 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0c44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c8ef29a1-5d62-45ad-b1bc-a0f77988a145_calibrate_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c8ef29a1-5d62-45ad-b1bc-a0f77988a145_calibrate_1123954_8.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6138ca77-60af-45e5-8503-b300e6cd0028_characterizeImage_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/6138ca77-60af-45e5-8503-b300e6cd0028_characterizeImage_1123954_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:41 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:41 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:41 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:42 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d487632e1552bf236fef049197abb49 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b713c253-8405-46ef-8f55-b8a1abf48350_calibrate_1123954_10.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/b713c253-8405-46ef-8f55-b8a1abf48350_calibrate_1123954_10.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f95e10a93188269542786cbef826b28 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/abe05d06-17ff-4b9d-a486-b40780616425_calibrate_1123956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/abe05d06-17ff-4b9d-a486-b40780616425_calibrate_1123956_39.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 284d0f80e86ba29f5c050fcbb29e2d99 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1440bc28-cffe-4c86-ba15-14355f359f0c_calibrate_1123958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/1440bc28-cffe-4c86-ba15-14355f359f0c_calibrate_1123958_32.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e125720385eb4c67516d61beccf14ce7 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8a748d1a-a7aa-4bd4-86de-dcabea4163cb_calibrate_1123958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8a748d1a-a7aa-4bd4-86de-dcabea4163cb_calibrate_1123958_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:42 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:42 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5cec7fca1931c9df732031c66b0a50c with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/378750f8-83f3-4a1f-97c4-0b4b945bfc8c_calibrate_1123958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/378750f8-83f3-4a1f-97c4-0b4b945bfc8c_calibrate_1123958_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:43 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1077cff816795d689032ffa4378ce73a with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/23ac7a5e-f363-459a-94e1-7c27611831a6_calibrate_1123956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/23ac7a5e-f363-459a-94e1-7c27611831a6_calibrate_1123956_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:43 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af79b4a361af586aa51ac71474fcc55c with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a60e78b-1786-46e1-9b1a-1d3cb4f16bae_calibrate_1123956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/4a60e78b-1786-46e1-9b1a-1d3cb4f16bae_calibrate_1123956_48.stderr parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe023f772bc85058019f3d60ff704fb5 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b59cf040-ea1e-4d16-9d03-ee025bc681c3_calibrate_1123956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b59cf040-ea1e-4d16-9d03-ee025bc681c3_calibrate_1123956_50.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 643ed61f90a48802373b85a09c9d8053 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/37fbc880-8d65-4e06-9d6c-9086c90fb905_calibrate_1123954_22.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/37fbc880-8d65-4e06-9d6c-9086c90fb905_calibrate_1123954_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:45 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0af5a77a148c76a9a145f7e570a5f1e with result from task 470 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:46 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/14430dba-2165-4fd2-931d-12e2e1001bb0_calibrate_1123956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/14430dba-2165-4fd2-931d-12e2e1001bb0_calibrate_1123956_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:46 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:46 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 624f0598a4b4bf88a7d24d650a2181a7 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/74dfebc2-094d-4f78-8084-ade00f89579e_calibrate_1123954_35.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/74dfebc2-094d-4f78-8084-ade00f89579e_calibrate_1123954_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:46 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6adaba0208f40f3a7ca3143ec9998667 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f2f0b687-9c49-4b6f-a367-8dcfe92744d4_calibrate_1123956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f2f0b687-9c49-4b6f-a367-8dcfe92744d4_calibrate_1123956_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:47 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:47 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:47 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3af96fbb40a505c91c664ab8cba16f37 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/80628c23-fcaa-45c9-a847-4f3e72a58dd6_calibrate_1123956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/80628c23-fcaa-45c9-a847-4f3e72a58dd6_calibrate_1123956_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:48 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:48 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:49 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d4684bf8c9dbbe14c697a42701c263d with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/1c366220-030f-4fb4-905c-a7b0d8d3366b_calibrate_1123954_12.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/1c366220-030f-4fb4-905c-a7b0d8d3366b_calibrate_1123954_12.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8d5a24c4753d80b87dca7c17bd4ba8e with result from task 584 parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:49 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/9c11d566-09ec-4ed5-b9bc-3cc7a808d330_calibrate_1123954_28.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/9c11d566-09ec-4ed5-b9bc-3cc7a808d330_calibrate_1123954_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:49 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:49 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:50 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:43:50 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec838f5399fa0bc0aaec9673f576de5 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a9ef69ba-9517-4aa3-9eac-823457fca4a7_calibrate_1123954_20.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a9ef69ba-9517-4aa3-9eac-823457fca4a7_calibrate_1123954_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 60 parsl.dataflow.strategy DEBUG: Executor multi has 60 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 60 parsl.dataflow.strategy DEBUG: Executor multi has 60 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 584b259c2a95b0e7ed7b3417a98561a6 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a94281db-1ba1-43ba-b4af-49a5eb87e9a9_calibrate_1123954_44.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a94281db-1ba1-43ba-b4af-49a5eb87e9a9_calibrate_1123954_44.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e755590a82e32f25ea4e5a5cf53ca453 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/89a2bed9-c700-48a4-8d8c-b68fb4b95a2a_calibrate_1123956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/89a2bed9-c700-48a4-8d8c-b68fb4b95a2a_calibrate_1123956_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 176 try 1 failed parsl.dataflow.dflow ERROR: Task 176 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 78d2bad458a54521e86137a027f978a5 with result from task 176 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8129db1-43aa-4693-9922-249958428fc0_calibrate_1123954_31.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22eb6c1ba4d7eea88a1d15a5d192f913 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/18f627dc-90e1-496a-9534-c6273042fd29_calibrate_1123954_34.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/18f627dc-90e1-496a-9534-c6273042fd29_calibrate_1123954_34.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032a7b14855fcb5fc7077f3f3f2f5721 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f4ab0173-657a-4ac1-9b38-8f21300ada2a_calibrate_1123956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/f4ab0173-657a-4ac1-9b38-8f21300ada2a_calibrate_1123956_22.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 875e9b762e93e5e39be33376e09bf4a5 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/e64be338-85cc-490b-b8c2-f0831ae1019b_calibrate_1123958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/e64be338-85cc-490b-b8c2-f0831ae1019b_calibrate_1123958_52.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 54 parsl.dataflow.strategy DEBUG: Executor multi has 54 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a64b9edcb64bf228ba2514a243e076c1 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/04ea1439-5700-4d64-a576-e4dccf41dc95_calibrate_1123958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/04ea1439-5700-4d64-a576-e4dccf41dc95_calibrate_1123958_47.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56b9a7a1c16c6f7be37581809e1a8605 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a331070f-50d7-4407-b2ae-d8c8e37e932f_calibrate_1123958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a331070f-50d7-4407-b2ae-d8c8e37e932f_calibrate_1123958_18.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d8798a95e43db3c792da28dde557945 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3df3c3a2-d840-4391-b6d5-3330497a5162_calibrate_1123956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3df3c3a2-d840-4391-b6d5-3330497a5162_calibrate_1123956_47.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee7495069fdf6482ac3708ce4504b6fe with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3e661fc1-bf1a-4ae1-ae27-588ed13b01b4_calibrate_1123956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3e661fc1-bf1a-4ae1-ae27-588ed13b01b4_calibrate_1123956_34.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd575a9d3e3038c4a1e459590704262f with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/11055724-d5e0-4de2-99a1-4cb4832afca6_calibrate_1123954_53.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/11055724-d5e0-4de2-99a1-4cb4832afca6_calibrate_1123954_53.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f806c1d56a2b1255a0be6f2d760c962c with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d2ac9435-caa1-4f9f-bf84-3ce8825f094d_calibrate_1123956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d2ac9435-caa1-4f9f-bf84-3ce8825f094d_calibrate_1123956_60.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 034e09c73a339df737155bf5e8c0c398 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/52cb2a01-e283-443a-beae-a275c4b2d842_calibrate_1123954_24.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/52cb2a01-e283-443a-beae-a275c4b2d842_calibrate_1123954_24.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 666235b7f1bf254e6fd9f376237c9c6b with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash c699c8a9469285ee3db3c3c6c84616f3 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1faa5c9-8699-438f-b9d2-f0de3091d575_calibrate_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1faa5c9-8699-438f-b9d2-f0de3091d575_calibrate_1123954_33.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9f472724-4b53-49b1-ae6f-f947e5687769_characterizeImage_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/9f472724-4b53-49b1-ae6f-f947e5687769_characterizeImage_1123954_33.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b937502ae5e6bcc50782580453e41ec with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a3fb4516-6a1c-4a83-af57-effd2eb4f495_calibrate_1123954_6.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/a3fb4516-6a1c-4a83-af57-effd2eb4f495_calibrate_1123954_6.stderr 2024-03-07 03:44:00 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:44:00 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:01 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:01 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:02 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:02 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb6b1134157037d459dd836fdc02977a with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/063c2b52-5bab-4458-b4e9-012292f5f426_calibrate_1123956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/063c2b52-5bab-4458-b4e9-012292f5f426_calibrate_1123956_23.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 356 try 1 failed parsl.dataflow.dflow ERROR: Task 356 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d3385e090b7b46f2bfa97d5bdeb2af82 with result from task 356 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5fb4ca62-9f39-4650-b427-9eed70fa6a73_calibrate_1123956_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:02 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:02 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 781753c15bcdd8eceb2640d88d0fad29 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8642def3-6bfb-49c9-bd0e-6d85d5a5450e_calibrate_1123954_27.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/8642def3-6bfb-49c9-bd0e-6d85d5a5450e_calibrate_1123954_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:03 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d335a52c2ee64fd7dcb232063440595f with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/798426ff-055b-431d-890a-125a32f99de6_calibrate_1123954_15.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/798426ff-055b-431d-890a-125a32f99de6_calibrate_1123954_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:03 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:03 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc4568618e4a2c79e856d29f2d96c383 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 0d7bf7093d941cf8981a464084a6bd6b parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1468873c51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3afae1a4-e3b9-4de0-8118-07a8fd2ddcdf_calibrate_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3afae1a4-e3b9-4de0-8118-07a8fd2ddcdf_calibrate_1123956_62.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c223182f-99a0-4aed-a159-a108ccf6028a_characterizeImage_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c223182f-99a0-4aed-a159-a108ccf6028a_characterizeImage_1123956_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:04 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8084c58517ad7a74e2a5cbc99ac8a3f9 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8eaf7ced-2eb1-4f5a-a359-6c51f0ba01f1_calibrate_1123958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/8eaf7ced-2eb1-4f5a-a359-6c51f0ba01f1_calibrate_1123958_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:04 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ad60b8cff0131934784d401b45cec61 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/140e54eb-6918-4c25-84dd-217473588fcf_calibrate_1123954_25.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/140e54eb-6918-4c25-84dd-217473588fcf_calibrate_1123954_25.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 333c48f3f03ce8fdc8384e0858cd6162 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfb7268e-836e-4d4d-a00c-4a3663626479_calibrate_1123958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfb7268e-836e-4d4d-a00c-4a3663626479_calibrate_1123958_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:04 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36b6f13e19d2f491442aaefe29487539 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8ab287a1-f86a-4e1a-ab85-cd920376f26e_calibrate_1123956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8ab287a1-f86a-4e1a-ab85-cd920376f26e_calibrate_1123956_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:05 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40c612fa8e51b134891ec13fee4b91f1 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d573f08e-9af0-4b9e-9c89-aa83d0540727_calibrate_1123954_54.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d573f08e-9af0-4b9e-9c89-aa83d0540727_calibrate_1123954_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:05 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:05 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:05 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:06 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9872d740224ee9ad9fed74d8306135c7 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/c0f52e32-3b23-4af5-84fc-f8806536ac81_calibrate_1123956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/c0f52e32-3b23-4af5-84fc-f8806536ac81_calibrate_1123956_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:06 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:06 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:07 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:07 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9574578c7fa7da0fc6a229a02663ed2 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e8c0d3ed-5d94-4c9c-b13e-b44935477b44_calibrate_1123954_37.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e8c0d3ed-5d94-4c9c-b13e-b44935477b44_calibrate_1123954_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:07 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e4f59454f04af86caa76c427c677505 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e5a78349-5eb3-44dc-be53-0846c589d2bd_calibrate_1123956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/e5a78349-5eb3-44dc-be53-0846c589d2bd_calibrate_1123956_28.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 537461d508ddc3d2b96e055f61359f14 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ebc6b77a-9823-4ef7-86ba-351aaebb84db_calibrate_1123956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/ebc6b77a-9823-4ef7-86ba-351aaebb84db_calibrate_1123956_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:08 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2122e2852cdf4eb68390b2bd752907e5 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2708764c-88dd-4621-89ee-392b83681923_calibrate_1123956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/2708764c-88dd-4621-89ee-392b83681923_calibrate_1123956_6.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4a7cce9fdce4f7738b4cfb451094690 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d81459be-ce21-4efe-ac93-a26c00619f12_calibrate_1123954_32.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d81459be-ce21-4efe-ac93-a26c00619f12_calibrate_1123954_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:08 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78351364b504afffb03cbac2f52a5a2f with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/51c50c6a-e675-4a2c-964d-ee9876fb0eff_calibrate_1123954_14.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/51c50c6a-e675-4a2c-964d-ee9876fb0eff_calibrate_1123954_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:09 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:09 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:09 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80050303745a0f4b56730859d6c09cab with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5d44d633-184e-4e0a-b259-f1f92c5936db_calibrate_1123958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/5d44d633-184e-4e0a-b259-f1f92c5936db_calibrate_1123958_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:10 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:10 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbf7ee67759e87ac8819f99a82192a2c with result from task 608 parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:10 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/7f320911-e7e1-4e13-9bf4-dc1dc57a8fd9_calibrate_1123958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/7f320911-e7e1-4e13-9bf4-dc1dc57a8fd9_calibrate_1123958_60.stderr parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e547113a9d3722b6c3d3423e61789b6 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8dcce1d1-e20a-4f0f-8049-909e532f014e_calibrate_1123956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/8dcce1d1-e20a-4f0f-8049-909e532f014e_calibrate_1123956_53.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 487a1cef26bbd096a39924d047732bcd with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5b7325bf-2e05-454f-bd88-bc4ed221f128_calibrate_1123956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/5b7325bf-2e05-454f-bd88-bc4ed221f128_calibrate_1123956_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:11 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:11 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:11 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:12 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:12 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97e5b46aff83aaaedf310307f5ef95df with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/33a51cf9-50eb-44ad-a7d2-ff7dc961b28e_calibrate_1123954_61.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/33a51cf9-50eb-44ad-a7d2-ff7dc961b28e_calibrate_1123954_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:12 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:13 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:13 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ca8a569772bc88308998cfe41433082 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a47e5e71-1600-499d-b7a5-819fd2299482_calibrate_1123958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/a47e5e71-1600-499d-b7a5-819fd2299482_calibrate_1123958_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:13 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:13 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:13 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:14 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:14 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bb5bc823e040ff7f0f64a24397fba38 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc0affe-281f-4e7d-89d1-165c36d8b9f6_calibrate_1123958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cdc0affe-281f-4e7d-89d1-165c36d8b9f6_calibrate_1123958_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:14 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:14 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:14 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3786280ec3208ff8d21f1e80a691efe9 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bffb2ee3-04c9-4aab-bd16-c9911cd4f79d_calibrate_1123956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/bffb2ee3-04c9-4aab-bd16-c9911cd4f79d_calibrate_1123956_59.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab05cc486844205fe02961cc37ddd06a with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/73762187-013f-4e6a-8050-96671123684e_calibrate_1123958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/73762187-013f-4e6a-8050-96671123684e_calibrate_1123958_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:14 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:15 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:15 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 304e0f467c493f87fe5369687c0fa184 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9683e906-03cd-4639-ad2c-d1503abf2b21_calibrate_1123958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/9683e906-03cd-4639-ad2c-d1503abf2b21_calibrate_1123958_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:15 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:15 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed32013fce11d75c1336dd528a7f3453 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/04d7eb24-c31c-42ce-8ac7-cafdfaf956dc_calibrate_1123954_56.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/04d7eb24-c31c-42ce-8ac7-cafdfaf956dc_calibrate_1123954_56.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7549a2e4293c1058bea5b84a4ab852cd with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/65ca3b6a-412a-4379-a87b-9c329c4f083a_calibrate_1123956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/65ca3b6a-412a-4379-a87b-9c329c4f083a_calibrate_1123956_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:17 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:17 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:17 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:17 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:18 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:18 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:18 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:18 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:18 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 18 parsl.dataflow.strategy DEBUG: Executor multi has 18 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 18 parsl.dataflow.strategy DEBUG: Executor multi has 18 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1273719bd752652c2054fff9fc44be3a with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d05cd663-e050-4a4a-a9b4-765c7387e974_calibrate_1123958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/d05cd663-e050-4a4a-a9b4-765c7387e974_calibrate_1123958_29.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3dbc2f7ed4c220e42eae8e8d2f15982 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cfc2015c-8db9-4ad9-ae62-bea358caff5c_calibrate_1123958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/cfc2015c-8db9-4ad9-ae62-bea358caff5c_calibrate_1123958_25.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0773a97ba6500c405d17f2235b08777 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f848cccf-5440-4bf9-a9b3-0e2427d86988_calibrate_1123958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/f848cccf-5440-4bf9-a9b3-0e2427d86988_calibrate_1123958_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 107 try 0 failed parsl.dataflow.dflow INFO: Task 107 marked for retry parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 729b2eb05b123063b2b85b9c463b7983 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca7c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 1 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cbe71f934415227c4a254535305c8ce with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 850f38e6a1a8e903584e87e3a3485071 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0f1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/5b5a4ba6-7013-4e0e-8390-4cf3d859197f_calibrate_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/5b5a4ba6-7013-4e0e-8390-4cf3d859197f_calibrate_1123954_11.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/23b33197-04ae-4b6d-9c8b-1522048c36a2_characterizeImage_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123954/23b33197-04ae-4b6d-9c8b-1522048c36a2_characterizeImage_1123954_11.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62479e0ec446c0319b62e6fab33b1563 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b73fcedb-51f9-42c9-9c05-e07c0cc8a606_calibrate_1123956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/b73fcedb-51f9-42c9-9c05-e07c0cc8a606_calibrate_1123956_13.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 14 parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9be790545c6c7ca45483c5d63d27308f with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8ac84be-4229-435b-bd6c-1f47992b2294_calibrate_1123954_59.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/d8ac84be-4229-435b-bd6c-1f47992b2294_calibrate_1123954_59.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 320e66b948ad10f7f495b00fc6e933ef with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3613447d-4853-4003-abb3-5f0137e0a792_calibrate_1123958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3613447d-4853-4003-abb3-5f0137e0a792_calibrate_1123958_21.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 035325d34b6f9ed82f903cdb86c40656 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/dee842f0-5517-43d3-9825-35db4d902284_calibrate_1123956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/dee842f0-5517-43d3-9825-35db4d902284_calibrate_1123956_8.stderr 2024-03-07 03:44:33 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:44:33 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 778753578739caab05073b66334dca97 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/186c4a1b-c904-4842-b566-e965de77073a_calibrate_1123956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/186c4a1b-c904-4842-b566-e965de77073a_calibrate_1123956_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:34 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:35 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 230 try 0 failed parsl.dataflow.dflow INFO: Task 230 marked for retry parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash e6c341e477a8239085ff1593f8b2048d parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688c0a0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 1 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:35 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:35 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:35 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:35 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:35 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:35 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:36 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:36 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:36 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:36 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:36 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:36 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:36 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:37 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a6ffd3f2b32c952e11f56fb492720c1 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/26a30aad-1ebc-4af5-ae8c-58c43707557d_calibrate_1123958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/26a30aad-1ebc-4af5-ae8c-58c43707557d_calibrate_1123958_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:38 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:38 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:38 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:38 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:38 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:38 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:38 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:39 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:40 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:40 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:40 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:40 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:40 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:40 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:40 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:44:41 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7358c250f7750e4fd9df4967ab532469 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c8ef29a1-5d62-45ad-b1bc-a0f77988a145_calibrate_1123954_8.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/c8ef29a1-5d62-45ad-b1bc-a0f77988a145_calibrate_1123954_8.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c699c8a9469285ee3db3c3c6c84616f3 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1faa5c9-8699-438f-b9d2-f0de3091d575_calibrate_1123954_33.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/e1faa5c9-8699-438f-b9d2-f0de3091d575_calibrate_1123954_33.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d7bf7093d941cf8981a464084a6bd6b with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3afae1a4-e3b9-4de0-8118-07a8fd2ddcdf_calibrate_1123956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/3afae1a4-e3b9-4de0-8118-07a8fd2ddcdf_calibrate_1123956_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:07 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:08 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:08 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:08 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:08 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:08 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:08 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:08 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:09 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:09 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:09 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:09 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:09 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:09 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:09 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:10 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:10 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:10 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:10 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:10 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:10 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:10 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:11 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:12 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:12 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:12 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:12 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:12 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:12 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:12 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:13 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 107 try 1 failed parsl.dataflow.dflow ERROR: Task 107 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.dataflow.memoization DEBUG: Storing app cache entry 729b2eb05b123063b2b85b9c463b7983 with result from task 107 parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:13 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/bfa73438-148b-401c-8679-411e8f8fffca_calibrate_1123958_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:13 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:13 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:13 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:13 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:13 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:14 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:15 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:15 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:15 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 850f38e6a1a8e903584e87e3a3485071 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/5b5a4ba6-7013-4e0e-8390-4cf3d859197f_calibrate_1123954_11.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123954/5b5a4ba6-7013-4e0e-8390-4cf3d859197f_calibrate_1123954_11.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af00cd310382dda5b192d46dfb10d433 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 1e4e43bbac8580bb9335430caf631782 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887218f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2c097470-9310-4b74-a28d-a72f8fe92734_calibrate_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2c097470-9310-4b74-a28d-a72f8fe92734_calibrate_1123958_35.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e5cebe92-bb9c-42a2-bd49-5689981d104a_characterizeImage_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123958/e5cebe92-bb9c-42a2-bd49-5689981d104a_characterizeImage_1123958_35.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 230 try 1 failed parsl.dataflow.dflow ERROR: Task 230 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e6c341e477a8239085ff1593f8b2048d with result from task 230 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/3a8996c6-1ebd-4cf9-9802-8f90e21c71e6_calibrate_1123958_61.stderr 2024-03-07 03:45:40 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:45:40 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:40 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:40 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:40 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:41 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:41 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:41 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:41 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:41 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:41 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:41 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:42 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:42 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:42 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:42 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:42 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:42 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:42 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:43 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:43 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:43 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:43 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:43 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:43 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:43 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:44 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:45 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:45 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:45 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:45 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:45 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:45 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:45 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:46 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:47 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:47 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:47 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:47 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:47 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:47 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:47 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:48 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:48 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:48 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:48 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:48 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:45:48 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 2024-03-07 03:46:13 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:46:13 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:14 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:14 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:14 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:14 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:14 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:14 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:14 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:15 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:16 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:16 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:16 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:16 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:16 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:16 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:16 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:17 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:17 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:17 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:17 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:17 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:17 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:17 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:18 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:19 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:19 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:19 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:19 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:19 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:19 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:19 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:20 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:21 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:21 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:21 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:21 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:21 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:21 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:21 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:22 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e4e43bbac8580bb9335430caf631782 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2c097470-9310-4b74-a28d-a72f8fe92734_calibrate_1123958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123958/2c097470-9310-4b74-a28d-a72f8fe92734_calibrate_1123958_35.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0850f13e5429f37a23fb6811ed0bf6da with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 84e4b6e476018cae37c9afea61a74c75 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14688ca56c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d5d54883-fa50-4836-af99-670c4d899524_calibrate_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d5d54883-fa50-4836-af99-670c4d899524_calibrate_1123956_37.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c52a730f-8427-4434-a78a-c64ba0c6e4ab_characterizeImage_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/c52a730f-8427-4434-a78a-c64ba0c6e4ab_characterizeImage_1123956_37.stderr 2024-03-07 03:46:47 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:46:47 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:47 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:47 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:47 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:47 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:47 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:48 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:49 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:49 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:49 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:49 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:49 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:49 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:49 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:50 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:51 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:52 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:52 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:52 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:52 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:52 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:52 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:52 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:53 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:53 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:53 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:53 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:53 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:53 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:53 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:54 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:56 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:46:57 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 2024-03-07 03:47:17 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:47:17 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:17 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:17 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:17 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:17 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:17 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:18 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:18 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:18 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:18 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:18 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:18 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:18 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:19 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:20 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:20 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:20 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:20 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:20 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:20 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:20 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:21 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:22 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:22 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:22 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:22 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:22 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:22 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:22 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:23 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:24 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:24 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:24 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:24 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:24 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:24 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:24 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:25 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:25 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:25 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84e4b6e476018cae37c9afea61a74c75 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d5d54883-fa50-4836-af99-670c4d899524_calibrate_1123956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/d5d54883-fa50-4836-af99-670c4d899524_calibrate_1123956_37.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c80a4c4839afccc6d908b42174a96921 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash b082bb7a5a80b93f51abaab25fd76df5 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47d7bb4c-757e-4461-a69d-8d5605ebc28b_characterizeImage_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/characterizeImage/1123956/47d7bb4c-757e-4461-a69d-8d5605ebc28b_characterizeImage_1123956_61.stderr 2024-03-07 03:47:50 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:47:50 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:50 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:50 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:50 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:51 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:52 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:52 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:52 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:52 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:52 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:52 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:52 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:53 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:54 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:54 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:54 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:54 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:54 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:54 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:54 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:55 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:55 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:55 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:55 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:55 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:55 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:55 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:56 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:57 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:57 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:57 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:57 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:57 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:57 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:57 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:58 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:58 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:58 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:58 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:47:58 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 2024-03-07 03:48:23 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:48:23 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:23 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:24 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:24 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:24 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:24 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:24 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:24 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:24 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:25 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:26 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:26 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:26 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:26 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:26 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:26 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:26 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:27 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:27 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:27 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:27 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:27 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:27 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:27 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:28 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:28 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:28 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:28 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:28 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:28 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:28 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:29 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:30 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:30 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:30 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:30 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:30 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:30 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:30 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:31 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:31 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:31 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:33 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:33 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:33 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:33 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:33 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 587 try 0 failed parsl.dataflow.dflow INFO: Task 587 marked for retry parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash b082bb7a5a80b93f51abaab25fd76df5 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x146887327ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 1 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stderr 2024-03-07 03:48:53 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:48:53 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:54 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:55 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:56 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:56 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:56 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:56 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:56 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:56 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:56 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:57 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:58 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:48:59 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:00 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:00 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:00 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:00 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:00 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:00 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:00 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:01 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:01 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:01 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:01 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:01 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:01 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 2024-03-07 03:49:26 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:49:26 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:27 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:28 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:29 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:29 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:29 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:29 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:29 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:29 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:29 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884260,16884261,16884262,16884263,16884264,16884265,16884266,16884267,16884268,16884269,16884270,16884271,16884272,16884273,16884274,16884275,16884276,16884277,16884278,16884279,16884280,16884281,16884282,16884283,16884284,16884285,16884286,16884287,16884288,16884289' parsl.providers.slurm.slurm DEBUG: squeue returned 16884285 R 16884286 R 16884287 R 16884288 R 16884289 R 16884275 R 16884276 R 16884277 R 16884278 R 16884279 R 16884280 R 16884281 R 16884282 R 16884283 R 16884284 R 16884271 R 16884272 R 16884273 R 16884274 R 16884268 R 16884269 R 16884270 R 16884264 R 16884265 R 16884266 R 16884267 R 16884261 R 16884262 R 16884263 R 16884260 R parsl.providers.slurm.slurm DEBUG: Updating job 16884285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884288 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884275 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884276 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884277 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884278 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884279 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884280 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884281 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884282 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884272 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884273 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884274 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884260 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:30 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:31 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:31 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:31 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884290,16884291,16884292,16884293,16884294' parsl.providers.slurm.slurm DEBUG: squeue returned 16884290 R 16884291 R 16884292 R 16884293 R 16884294 R parsl.providers.slurm.slurm DEBUG: Updating job 16884290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884294 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:31 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:31 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:31 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:31 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:32 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:33 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:33 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:33 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:33 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:33 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:33 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:33 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:34 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884295,16884296,16884297,16884298,16884299,16884300,16884301,16884302,16884303,16884304,16884305,16884306,16884307,16884308,16884309,16884310,16884311,16884312,16884313,16884314' parsl.providers.slurm.slurm DEBUG: squeue returned 16884314 R 16884309 R 16884310 R 16884311 R 16884312 R 16884313 R 16884304 R 16884305 R 16884306 R 16884307 R 16884308 R 16884295 R 16884296 R 16884297 R 16884298 R 16884299 R 16884300 R 16884301 R 16884302 R 16884303 R parsl.providers.slurm.slurm DEBUG: Updating job 16884314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884303 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:34 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:34 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:34 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:34 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:34 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING 2024-03-07 03:49:34 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16884315,16884316,16884317,16884318,16884319,16884320' parsl.providers.slurm.slurm DEBUG: squeue returned 16884315 R 16884316 R 16884317 R 16884318 R 16884319 R 16884320 R parsl.providers.slurm.slurm DEBUG: Updating job 16884315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16884320 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 62, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 62/0 running/pending blocks, and 105 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22439275932688 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 587 try 1 failed parsl.dataflow.dflow ERROR: Task 587 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b082bb7a5a80b93f51abaab25fd76df5 with result from task 587 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 608 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Tasks in state States.failed: 71 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 65 parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/submit/DEEP/20220821/B1m/science#step1/20240307T033644Z/logs/calibrate/1123956/537f7227-6040-49db-b4ce-8a23f37f479f_calibrate_1123956_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=62, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 62 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3bf62e684b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3bf62e684b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f54fe493f9e4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f54fe493f9e4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5333cf31c8ce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5333cf31c8ce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d5cbd56cad6b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d5cbd56cad6b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19159ebbdd70 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19159ebbdd70 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b6681b503aae parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b6681b503aae parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 53e6c2b06092 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 53e6c2b06092 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ccfdb207854 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ccfdb207854 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0097ff76e7a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0097ff76e7a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ea3c50956b10 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ea3c50956b10 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8cca7fa93a93 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8cca7fa93a93 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e63871a64c67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e63871a64c67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f8085c8fc6ec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f8085c8fc6ec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4bd0b169da2b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4bd0b169da2b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51446bbf8252 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51446bbf8252 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 46f1e59ac363 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 46f1e59ac363 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e1b807e2e198 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e1b807e2e198 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c47498b31db5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c47498b31db5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7b4fe3b7b6cf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7b4fe3b7b6cf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 058a4e89eae6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 058a4e89eae6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 727c03d7526b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 727c03d7526b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6cdfb185ac93 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6cdfb185ac93 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 754367074546 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 754367074546 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3ba363ec1af parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3ba363ec1af parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2fb3c102d575 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2fb3c102d575 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: def77cfa832d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: def77cfa832d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27634a80d1b5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27634a80d1b5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2b52e1aa38b5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2b52e1aa38b5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8cb3d84f66b5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8cb3d84f66b5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e7daf5f2bf21 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e7daf5f2bf21 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76762cc3ddba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76762cc3ddba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fefc8800aeb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fefc8800aeb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2fae6c49c6ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2fae6c49c6ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ca96fdbc8037 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ca96fdbc8037 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fba0ac88ecaa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fba0ac88ecaa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e2a25f2864c1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e2a25f2864c1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2747f046d629 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2747f046d629 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 38d5f97c6546 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 38d5f97c6546 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ff7e8eb15f7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ff7e8eb15f7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e59f6a084f8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e59f6a084f8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9af501473fd7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9af501473fd7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6a48a83fb2f0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6a48a83fb2f0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f384871e97f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f384871e97f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9ad903e7c92f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9ad903e7c92f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 553a217942c2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 553a217942c2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c5aa51ec1194 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c5aa51ec1194 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d5bdb87142e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d5bdb87142e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b0673543b0ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b0673543b0ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce544f064371 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce544f064371 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78d723e12579 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78d723e12579 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a996bd7b0c72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a996bd7b0c72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5db3e0ad4cbf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5db3e0ad4cbf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 448423f1b659 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 448423f1b659 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 007453a85edc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 007453a85edc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6e5a9b3117f3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6e5a9b3117f3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bcbe8ad018ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bcbe8ad018ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a61e11830bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a61e11830bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db11914b2f18 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db11914b2f18 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9a07e0b745f0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9a07e0b745f0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db8885dc9e99 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db8885dc9e99 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c28322c02508 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c28322c02508 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 271ad3f4c9bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 271ad3f4c9bf 2024-03-07 03:49:43 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:49:43 proc_lsst.multi:201 [INFO] cancelling 4364 on provider local proc_lsst.multi INFO: cancelling 4364 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 4364 2024-03-07 03:49:43 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint 2024-03-07 03:49:43 proc_lsst.multi:201 [INFO] cancelling 16884315 on provider checkpoint proc_lsst.multi INFO: cancelling 16884315 on provider checkpoint 2024-03-07 03:49:43 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro 2024-03-07 03:49:43 proc_lsst.multi:201 [INFO] cancelling 16884281 on provider astro proc_lsst.multi INFO: cancelling 16884281 on provider astro 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884266 on provider astro proc_lsst.multi INFO: cancelling 16884266 on provider astro 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884290 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884290 on provider escience_rtx6k 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884275 on provider astro proc_lsst.multi INFO: cancelling 16884275 on provider astro 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884319 on provider checkpoint proc_lsst.multi INFO: cancelling 16884319 on provider checkpoint 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884288 on provider astro proc_lsst.multi INFO: cancelling 16884288 on provider astro 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884313 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884313 on provider escience_a40 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884286 on provider astro proc_lsst.multi INFO: cancelling 16884286 on provider astro 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884283 on provider astro proc_lsst.multi INFO: cancelling 16884283 on provider astro 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884289 on provider astro proc_lsst.multi INFO: cancelling 16884289 on provider astro 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884317 on provider checkpoint proc_lsst.multi INFO: cancelling 16884317 on provider checkpoint 2024-03-07 03:49:44 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint 2024-03-07 03:49:44 proc_lsst.multi:201 [INFO] cancelling 16884316 on provider checkpoint proc_lsst.multi INFO: cancelling 16884316 on provider checkpoint 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884303 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884303 on provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884280 on provider astro proc_lsst.multi INFO: cancelling 16884280 on provider astro 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884302 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884302 on provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884295 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884295 on provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884267 on provider astro proc_lsst.multi INFO: cancelling 16884267 on provider astro 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884292 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884292 on provider escience_rtx6k 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884299 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884299 on provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884262 on provider astro proc_lsst.multi INFO: cancelling 16884262 on provider astro 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884318 on provider checkpoint proc_lsst.multi INFO: cancelling 16884318 on provider checkpoint 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884306 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884306 on provider escience_a40 2024-03-07 03:49:45 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k 2024-03-07 03:49:45 proc_lsst.multi:201 [INFO] cancelling 16884293 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884293 on provider escience_rtx6k 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884291 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884291 on provider escience_rtx6k 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884264 on provider astro proc_lsst.multi INFO: cancelling 16884264 on provider astro 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884261 on provider astro proc_lsst.multi INFO: cancelling 16884261 on provider astro 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884320 on provider checkpoint proc_lsst.multi INFO: cancelling 16884320 on provider checkpoint 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884282 on provider astro proc_lsst.multi INFO: cancelling 16884282 on provider astro 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884301 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884301 on provider escience_a40 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884287 on provider astro proc_lsst.multi INFO: cancelling 16884287 on provider astro 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884273 on provider astro proc_lsst.multi INFO: cancelling 16884273 on provider astro 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884278 on provider astro proc_lsst.multi INFO: cancelling 16884278 on provider astro 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884296 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884296 on provider escience_a40 2024-03-07 03:49:46 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 2024-03-07 03:49:46 proc_lsst.multi:201 [INFO] cancelling 16884304 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884304 on provider escience_a40 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884279 on provider astro proc_lsst.multi INFO: cancelling 16884279 on provider astro 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884260 on provider astro proc_lsst.multi INFO: cancelling 16884260 on provider astro 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884300 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884300 on provider escience_a40 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884263 on provider astro proc_lsst.multi INFO: cancelling 16884263 on provider astro 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884276 on provider astro proc_lsst.multi INFO: cancelling 16884276 on provider astro 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884311 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884311 on provider escience_a40 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884307 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884307 on provider escience_a40 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884274 on provider astro proc_lsst.multi INFO: cancelling 16884274 on provider astro 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884270 on provider astro proc_lsst.multi INFO: cancelling 16884270 on provider astro 2024-03-07 03:49:47 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro 2024-03-07 03:49:47 proc_lsst.multi:201 [INFO] cancelling 16884271 on provider astro proc_lsst.multi INFO: cancelling 16884271 on provider astro 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884268 on provider astro proc_lsst.multi INFO: cancelling 16884268 on provider astro 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884265 on provider astro proc_lsst.multi INFO: cancelling 16884265 on provider astro 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884298 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884298 on provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884277 on provider astro proc_lsst.multi INFO: cancelling 16884277 on provider astro 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884310 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884310 on provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884312 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884312 on provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884297 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884297 on provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 2024-03-07 03:49:48 proc_lsst.multi:201 [INFO] cancelling 16884305 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884305 on provider escience_a40 2024-03-07 03:49:49 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro 2024-03-07 03:49:49 proc_lsst.multi:201 [INFO] cancelling 16884269 on provider astro proc_lsst.multi INFO: cancelling 16884269 on provider astro 2024-03-07 03:49:49 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro 2024-03-07 03:49:49 proc_lsst.multi:201 [INFO] cancelling 16884284 on provider astro proc_lsst.multi INFO: cancelling 16884284 on provider astro 2024-03-07 03:49:49 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 2024-03-07 03:49:49 proc_lsst.multi:201 [INFO] cancelling 16884309 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884309 on provider escience_a40 2024-03-07 03:49:49 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro 2024-03-07 03:49:49 proc_lsst.multi:201 [INFO] cancelling 16884272 on provider astro proc_lsst.multi INFO: cancelling 16884272 on provider astro 2024-03-07 03:49:49 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 2024-03-07 03:49:49 proc_lsst.multi:201 [INFO] cancelling 16884308 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884308 on provider escience_a40 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884285 on provider astro proc_lsst.multi INFO: cancelling 16884285 on provider astro 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884314 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884314 on provider escience_a40 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884294 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884294 on provider escience_rtx6k parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 03:49:50 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 03:49:50 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16884315', '16884316', '16884317', '16884318', '16884319', '16884320', '16884260', '16884261', '16884262', '16884263', '16884264', '16884265', '16884266', '16884267', '16884268', '16884269', '16884270', '16884271', '16884272', '16884273', '16884274', '16884275', '16884276', '16884277', '16884278', '16884279', '16884280', '16884281', '16884282', '16884283', '16884284', '16884285', '16884286', '16884287', '16884288', '16884289', '16884295', '16884296', '16884297', '16884298', '16884299', '16884300', '16884301', '16884302', '16884303', '16884304', '16884305', '16884306', '16884307', '16884308', '16884309', '16884310', '16884311', '16884312', '16884313', '16884314', '16884290', '16884291', '16884292', '16884293', '16884294', '4364'] proc_lsst.multi INFO: new jobs since last cancel ['16884315', '16884316', '16884317', '16884318', '16884319', '16884320', '16884260', '16884261', '16884262', '16884263', '16884264', '16884265', '16884266', '16884267', '16884268', '16884269', '16884270', '16884271', '16884272', '16884273', '16884274', '16884275', '16884276', '16884277', '16884278', '16884279', '16884280', '16884281', '16884282', '16884283', '16884284', '16884285', '16884286', '16884287', '16884288', '16884289', '16884295', '16884296', '16884297', '16884298', '16884299', '16884300', '16884301', '16884302', '16884303', '16884304', '16884305', '16884306', '16884307', '16884308', '16884309', '16884310', '16884311', '16884312', '16884313', '16884314', '16884290', '16884291', '16884292', '16884293', '16884294', '4364'] 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884315 in provider checkpoint proc_lsst.multi INFO: found job 16884315 in provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884315 on provider checkpoint proc_lsst.multi INFO: cancelling 16884315 on provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884316 in provider checkpoint proc_lsst.multi INFO: found job 16884316 in provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884316 on provider checkpoint proc_lsst.multi INFO: cancelling 16884316 on provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884317 in provider checkpoint proc_lsst.multi INFO: found job 16884317 in provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884317 on provider checkpoint proc_lsst.multi INFO: cancelling 16884317 on provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884318 in provider checkpoint proc_lsst.multi INFO: found job 16884318 in provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884318 on provider checkpoint proc_lsst.multi INFO: cancelling 16884318 on provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:146 [INFO] found job 16884319 in provider checkpoint proc_lsst.multi INFO: found job 16884319 in provider checkpoint 2024-03-07 03:49:50 proc_lsst.multi:201 [INFO] cancelling 16884319 on provider checkpoint proc_lsst.multi INFO: cancelling 16884319 on provider checkpoint 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884320 in provider checkpoint proc_lsst.multi INFO: found job 16884320 in provider checkpoint 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884320 on provider checkpoint proc_lsst.multi INFO: cancelling 16884320 on provider checkpoint 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884260 in provider astro proc_lsst.multi INFO: found job 16884260 in provider astro 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884260 on provider astro proc_lsst.multi INFO: cancelling 16884260 on provider astro 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884261 in provider astro proc_lsst.multi INFO: found job 16884261 in provider astro 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884261 on provider astro proc_lsst.multi INFO: cancelling 16884261 on provider astro 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884262 in provider astro proc_lsst.multi INFO: found job 16884262 in provider astro 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884262 on provider astro proc_lsst.multi INFO: cancelling 16884262 on provider astro 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884263 in provider astro proc_lsst.multi INFO: found job 16884263 in provider astro 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884263 on provider astro proc_lsst.multi INFO: cancelling 16884263 on provider astro 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884264 in provider astro proc_lsst.multi INFO: found job 16884264 in provider astro 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884264 on provider astro proc_lsst.multi INFO: cancelling 16884264 on provider astro 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884265 in provider astro proc_lsst.multi INFO: found job 16884265 in provider astro 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884265 on provider astro proc_lsst.multi INFO: cancelling 16884265 on provider astro 2024-03-07 03:49:51 proc_lsst.multi:146 [INFO] found job 16884266 in provider astro proc_lsst.multi INFO: found job 16884266 in provider astro 2024-03-07 03:49:51 proc_lsst.multi:201 [INFO] cancelling 16884266 on provider astro proc_lsst.multi INFO: cancelling 16884266 on provider astro 2024-03-07 03:49:52 proc_lsst.multi:146 [INFO] found job 16884267 in provider astro proc_lsst.multi INFO: found job 16884267 in provider astro 2024-03-07 03:49:52 proc_lsst.multi:201 [INFO] cancelling 16884267 on provider astro proc_lsst.multi INFO: cancelling 16884267 on provider astro 2024-03-07 03:49:52 proc_lsst.multi:146 [INFO] found job 16884268 in provider astro proc_lsst.multi INFO: found job 16884268 in provider astro 2024-03-07 03:49:52 proc_lsst.multi:201 [INFO] cancelling 16884268 on provider astro proc_lsst.multi INFO: cancelling 16884268 on provider astro 2024-03-07 03:49:52 proc_lsst.multi:146 [INFO] found job 16884269 in provider astro proc_lsst.multi INFO: found job 16884269 in provider astro 2024-03-07 03:49:52 proc_lsst.multi:201 [INFO] cancelling 16884269 on provider astro proc_lsst.multi INFO: cancelling 16884269 on provider astro 2024-03-07 03:49:52 proc_lsst.multi:146 [INFO] found job 16884270 in provider astro proc_lsst.multi INFO: found job 16884270 in provider astro 2024-03-07 03:49:52 proc_lsst.multi:201 [INFO] cancelling 16884270 on provider astro proc_lsst.multi INFO: cancelling 16884270 on provider astro 2024-03-07 03:49:52 proc_lsst.multi:146 [INFO] found job 16884271 in provider astro proc_lsst.multi INFO: found job 16884271 in provider astro 2024-03-07 03:49:52 proc_lsst.multi:201 [INFO] cancelling 16884271 on provider astro proc_lsst.multi INFO: cancelling 16884271 on provider astro 2024-03-07 03:49:52 proc_lsst.multi:146 [INFO] found job 16884272 in provider astro proc_lsst.multi INFO: found job 16884272 in provider astro 2024-03-07 03:49:52 proc_lsst.multi:201 [INFO] cancelling 16884272 on provider astro proc_lsst.multi INFO: cancelling 16884272 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884273 in provider astro proc_lsst.multi INFO: found job 16884273 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884273 on provider astro proc_lsst.multi INFO: cancelling 16884273 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884274 in provider astro proc_lsst.multi INFO: found job 16884274 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884274 on provider astro proc_lsst.multi INFO: cancelling 16884274 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884275 in provider astro proc_lsst.multi INFO: found job 16884275 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884275 on provider astro proc_lsst.multi INFO: cancelling 16884275 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884276 in provider astro proc_lsst.multi INFO: found job 16884276 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884276 on provider astro proc_lsst.multi INFO: cancelling 16884276 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884277 in provider astro proc_lsst.multi INFO: found job 16884277 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884277 on provider astro proc_lsst.multi INFO: cancelling 16884277 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884278 in provider astro proc_lsst.multi INFO: found job 16884278 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884278 on provider astro proc_lsst.multi INFO: cancelling 16884278 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884279 in provider astro proc_lsst.multi INFO: found job 16884279 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884279 on provider astro proc_lsst.multi INFO: cancelling 16884279 on provider astro 2024-03-07 03:49:53 proc_lsst.multi:146 [INFO] found job 16884280 in provider astro proc_lsst.multi INFO: found job 16884280 in provider astro 2024-03-07 03:49:53 proc_lsst.multi:201 [INFO] cancelling 16884280 on provider astro proc_lsst.multi INFO: cancelling 16884280 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884281 in provider astro proc_lsst.multi INFO: found job 16884281 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884281 on provider astro proc_lsst.multi INFO: cancelling 16884281 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884282 in provider astro proc_lsst.multi INFO: found job 16884282 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884282 on provider astro proc_lsst.multi INFO: cancelling 16884282 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884283 in provider astro proc_lsst.multi INFO: found job 16884283 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884283 on provider astro proc_lsst.multi INFO: cancelling 16884283 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884284 in provider astro proc_lsst.multi INFO: found job 16884284 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884284 on provider astro proc_lsst.multi INFO: cancelling 16884284 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884285 in provider astro proc_lsst.multi INFO: found job 16884285 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884285 on provider astro proc_lsst.multi INFO: cancelling 16884285 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884286 in provider astro proc_lsst.multi INFO: found job 16884286 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884286 on provider astro proc_lsst.multi INFO: cancelling 16884286 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884287 in provider astro proc_lsst.multi INFO: found job 16884287 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884287 on provider astro proc_lsst.multi INFO: cancelling 16884287 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884288 in provider astro proc_lsst.multi INFO: found job 16884288 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884288 on provider astro proc_lsst.multi INFO: cancelling 16884288 on provider astro 2024-03-07 03:49:54 proc_lsst.multi:146 [INFO] found job 16884289 in provider astro proc_lsst.multi INFO: found job 16884289 in provider astro 2024-03-07 03:49:54 proc_lsst.multi:201 [INFO] cancelling 16884289 on provider astro proc_lsst.multi INFO: cancelling 16884289 on provider astro 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884295 in provider escience_a40 proc_lsst.multi INFO: found job 16884295 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884295 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884295 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884296 in provider escience_a40 proc_lsst.multi INFO: found job 16884296 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884296 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884296 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884297 in provider escience_a40 proc_lsst.multi INFO: found job 16884297 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884297 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884297 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884298 in provider escience_a40 proc_lsst.multi INFO: found job 16884298 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884298 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884298 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884299 in provider escience_a40 proc_lsst.multi INFO: found job 16884299 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884299 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884299 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884300 in provider escience_a40 proc_lsst.multi INFO: found job 16884300 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884300 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884300 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884301 in provider escience_a40 proc_lsst.multi INFO: found job 16884301 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884301 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884301 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884302 in provider escience_a40 proc_lsst.multi INFO: found job 16884302 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884302 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884302 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884303 in provider escience_a40 proc_lsst.multi INFO: found job 16884303 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884303 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884303 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884304 in provider escience_a40 proc_lsst.multi INFO: found job 16884304 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884304 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884304 on provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:146 [INFO] found job 16884305 in provider escience_a40 proc_lsst.multi INFO: found job 16884305 in provider escience_a40 2024-03-07 03:49:55 proc_lsst.multi:201 [INFO] cancelling 16884305 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884305 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884306 in provider escience_a40 proc_lsst.multi INFO: found job 16884306 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884306 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884306 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884307 in provider escience_a40 proc_lsst.multi INFO: found job 16884307 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884307 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884307 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884308 in provider escience_a40 proc_lsst.multi INFO: found job 16884308 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884308 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884308 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884309 in provider escience_a40 proc_lsst.multi INFO: found job 16884309 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884309 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884309 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884310 in provider escience_a40 proc_lsst.multi INFO: found job 16884310 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884310 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884310 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884311 in provider escience_a40 proc_lsst.multi INFO: found job 16884311 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884311 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884311 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884312 in provider escience_a40 proc_lsst.multi INFO: found job 16884312 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884312 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884312 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884313 in provider escience_a40 proc_lsst.multi INFO: found job 16884313 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884313 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884313 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884314 in provider escience_a40 proc_lsst.multi INFO: found job 16884314 in provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884314 on provider escience_a40 proc_lsst.multi INFO: cancelling 16884314 on provider escience_a40 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884290 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884290 in provider escience_rtx6k 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884290 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884290 on provider escience_rtx6k 2024-03-07 03:49:56 proc_lsst.multi:146 [INFO] found job 16884291 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884291 in provider escience_rtx6k 2024-03-07 03:49:56 proc_lsst.multi:201 [INFO] cancelling 16884291 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884291 on provider escience_rtx6k 2024-03-07 03:49:57 proc_lsst.multi:146 [INFO] found job 16884292 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884292 in provider escience_rtx6k 2024-03-07 03:49:57 proc_lsst.multi:201 [INFO] cancelling 16884292 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884292 on provider escience_rtx6k 2024-03-07 03:49:57 proc_lsst.multi:146 [INFO] found job 16884293 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884293 in provider escience_rtx6k 2024-03-07 03:49:57 proc_lsst.multi:201 [INFO] cancelling 16884293 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884293 on provider escience_rtx6k 2024-03-07 03:49:57 proc_lsst.multi:146 [INFO] found job 16884294 in provider escience_rtx6k proc_lsst.multi INFO: found job 16884294 in provider escience_rtx6k 2024-03-07 03:49:57 proc_lsst.multi:201 [INFO] cancelling 16884294 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16884294 on provider escience_rtx6k 2024-03-07 03:49:57 proc_lsst.multi:146 [INFO] found job 4364 in provider local proc_lsst.multi INFO: found job 4364 in provider local 2024-03-07 03:49:57 proc_lsst.multi:201 [INFO] cancelling 4364 on provider local proc_lsst.multi INFO: cancelling 4364 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 4364 parsl.providers.local.local WARNING: Failed to kill PID: 4364 and child processes on local 2024-03-07 03:49:58 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 774.0390 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220821_B1m_science#step1_20240307T033644Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 774.0449 seconds; current memory usage: 0.378 Gibyte, delta: 0.009 Gibyte, peak delta: 0.009 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 791.0305 seconds; current memory usage: 0.378 Gibyte, delta: 0.240 Gibyte, peak delta: 0.240 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.378 Gibyte (main), 0.378 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220821_B1m_science#step1_20240307T033644Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up